Evaluation · Training · Code

Seed Diffusion

Free while signed in. Answers cite the passages they came from.

First page

The curator’s take

Researchers from ByteDance and Tsinghua University introduce Seed Diffusion Preview, a discrete-state diffusion-based LLM optimized for code generation, achieving 2,146 tokens/sec on H20 GPUs while maintaining competitive benchmark performance. Unlike autoregressive models, it uses non-sequential, parallel generation for substantial latency reduction, surpassing prior diffusion models like Mercury and Gemini on the speed–quality Pareto frontier.

Key points

Two-Stage Curriculum (TSC) – Combines mask-based forward corruption (80% of training) with an edit-based process (20%) to improve calibration and reduce repetition. Avoids “carry-over unmasking” to prevent overconfidence and enable self-correction.

Constrained-order training – After pretraining, the model is fine-tuned on high-quality generation trajectories distilled from itself, limiting to more optimal token orders for better alignment with language structure.

On-policy diffusion learning – Optimizes for fewer generation steps without severe quality drop, using a verifier-guided objective to maintain correctness and stability.

Block-level parallel inference – Employs a semi-autoregressive scheme with KV-caching, generating tokens in blocks for speed while preserving quality. Infrastructure optimizations further improve throughput.

Strong benchmark results – Competitive with top code LMs on HumanEval, MBPP, BigCodeBench, LiveCodeBench, MBXP, NaturalCodeBench, and excels at editing tasks (Aider, CanItEdit).

Every Monday

Get next week’s papers.

The same picks and the same summaries, in your inbox. Free, and 176 issues deep.

Subscribe on Substack