This repository provides DiffuCoder, a diffusion-based large language model for code generation, addressing limitations in existing diffusion LLMs' generation patterns and post-training strategies. It targets researchers and developers in AI code generation, offering potentially faster generation than autoregressive models and improved performance through novel techniques.

How It Works

DiffuCoder builds upon Masked Denoising Models (MDMs) and diffusion LLMs (dLLMs), investigating how their generation patterns differ from autoregressive models. It introduces a new metric, the "autoregressiveness score," to quantify causal patterns during generation. A key innovation is Coupled-GRPO, a post-training method that addresses inefficiencies in per-timestep loss computation by using a coupled-sampling scheme. This scheme ensures all tokens receive a learning signal and improves probability estimates by evaluating tokens in partially-masked contexts, offering better accuracy with modest computational overhead.

Quick Start & Requirements

• Installation: Clone huggingface/open-r1, merge provided files, and set up the environment using conda and pip (e.g., pip install vllm==0.8.4, flash-attn==2.8.0.post1, setuptools, .[code]).
• Prerequisites: Python 3.11, CUDA, E2B API token (for code sandbox), wandb account.
• Data Preparation: Requires TIGER-Lab/AceCode-89K dataset for GRPO training.
• Resources: Training requires a code sandbox and wandb for logging. Inference requires a CUDA-enabled GPU.
• Links: Open-R1, Huggingface Models, Paper

Highlighted Details

• Models are available on Huggingface (Base, Instruct, cpGRPO).
• Supports inference with configurable TOKEN_PER_STEP for performance/speed trade-offs.
• Implements Coupled-GRPO for improved diffusion LLM training.
• Code evaluation leverages Qwen2.5-Coder.

Maintenance & Community

• Project is associated with Apple and research contributions from multiple authors.
• Updates mention ongoing MLX support for Apple Silicon.
• Code is based on huggingface/open-r1 and LLaMA-Factory.

Licensing & Compatibility

• The README does not explicitly state a license. The underlying open-r1 project is Apache 2.0 licensed. Compatibility for commercial use or closed-source linking requires clarification.

Limitations & Caveats

MLX support for Apple Silicon is listed as "in progress" as of June 2025, indicating potential limitations for users on that platform. The specific license for this repository is not clearly stated, which could impact commercial adoption.