Seed Diffusion
Researchers from ByteDance and Tsinghua University introduce Seed Diffusion Preview, a discrete-state diffusion-based LLM optimized for code generation, achieving 2,146 tokens/sec on H20 GPUs while maintaining competitive benchmark performance. Unlike autoregressive models, it uses non-sequential, parallel generation for substantial latency reduction, surpassing prior diffusion models like Mercury and Gemini on the speedâquality Pareto frontier.
Ask this paper
Two-Stage Curriculum (TSC) â Combines mask-based forward corruption (80% of training) with an edit-based process (20%) to improve calibration and reduce repetition. Avoids âcarry-over unmaskingâ to prevent overconfidence and enable self-correction.
Constrained-order training â After pretraining, the model is fine-tuned on high-quality generation trajectories distilled from itself, limiting to more optimal token orders for better alignment with language structure.
On-policy diffusion learning â Optimizes for fewer generation steps without severe quality drop, using a verifier-guided objective to maintain correctness and stability.
Block-level parallel inference â Employs a semi-autoregressive scheme with KV-caching, generating tokens in blocks for speed while preserving quality. Infrastructure optimizations further improve throughput.
Strong benchmark results â Competitive with top code LMs on HumanEval, MBPP, BigCodeBench, LiveCodeBench, MBXP, NaturalCodeBench, and excels at editing tasks (Aider, CanItEdit).