Diffusion LMs are Super Data Learners
Free while signed in. Answers cite the passages they came from.
Researchers from NUS, Sea AI Lab, StepFun, and collaborators demonstrate that diffusion language models (DLMs) consistently outperform autoregressive models when unique training data is limited, revealing a systematic "crossover" phenomenon where DLMs extract 3x more value per unique token through multi-epoch training. - **Crossover phenomenon**: Under limited unique data, DLMs surpass AR models with more epochs. Crossover timing shifts based on data quantity (more delays it), data quality (higher delays it), and model size (larger triggers earlier). - **Three compounding advantages**: (1) Any-order modeling enabling 2^L corruption patterns vs AR's L prefixes, (2) super-dense compute through iterative bidirectional denoising (>100x training FLOPs), (3) built-in Monte Carlo augmentation via masked sequence expectations. - **Dramatic efficiency at scale**: 1.7B DLM on 10B Python tokens for ~150 epochs (1.5T total) surpasses AR baseline on MBPP/MBPP+. 1B DLM achieves 56% HellaSwag and 33% MMLU using only 1B tokens repeated 480 epochs. - **Data vs compute trade-off**: DLMs achieve >3x data efficiency but require >100x training FLOPs and 16-4700x inference FLOPs, optimal when high-quality data is the primary constraint. - **Validation loss decoupling**: Rising validation cross-entropy doesn't imply degraded performance. Models continue improving on HellaSwag, MMLU, MBPP, HumanEval as relative NLL gaps widen consistently. - **Ablation insights**: Noise injection in AR inputs (10-90% masking) or dropout improves data-constrained performance but falls short of DLMs. Sparse AR degrades badly while DLM MoEs benefit consistently, confirming super-density advantage.
Crossover phenomenon: Under limited unique data, DLMs surpass AR models with more epochs. Crossover timing shifts based on data quantity (more delays it), data quality (higher delays it), and model size (larger triggers earlier).
Three compounding advantages: (1) Any-order modeling enabling 2^L corruption patterns vs AR’s L prefixes, (2) super-dense compute through iterative bidirectional denoising (>100x training FLOPs), (3) built-in Monte Carlo augmentation via masked sequence expectations.
Dramatic efficiency at scale: 1.7B DLM on 10B Python tokens for ~150 epochs (1.5T total) surpasses AR baseline on MBPP/MBPP+. 1B DLM achieves 56% HellaSwag and 33% MMLU using only 1B tokens repeated 480 epochs.
Data vs compute trade-off: DLMs achieve >3x data efficiency but require >100x training FLOPs and 16-4700x inference FLOPs, optimal when high-quality data is the primary constraint.
Validation loss decoupling: Rising validation cross-entropy doesn’t imply degraded performance. Models continue improving on HellaSwag, MMLU, MBPP, and HumanEval as relative NLL gaps widen consistently.
Ablation insights: Noise injection in AR inputs (10-90% masking) or dropout improves data-constrained performance but falls short of DLMs. Sparse AR degrades badly while DLM MoEs benefit consistently, confirming the super-density advantage.
Abstract
Under strictly controlled pre-training settings, we observe a Crossover: when unique data is limited, diffusion language models (DLMs) consistently surpass autoregressive (AR) models by training for more epochs. The crossover shifts later with more or higher-quality data, earlier with larger models, and persists across dense and sparse architectures. We attribute the gains to three compounding factors: (1) any-order modeling, (2) super-dense compute from iterative bidirectional denoising, and (3) built-in Monte Carlo augmentation; input or parameter noise improves AR under data constraint but cannot close the gap. At scale, a 1.7B DLM trained with a ~1.5T-token compute budget on 10B unique Python tokens overtakes an AR coder trained with strictly matched settings. In addition, a 1B-parameter DLM achieves > 56% accuracy on HellaSwag and > 33% on MMLU using only 1B tokens, without any special tricks, just by repeating standard pre-training data. We also show that rising validation cross-entropy does not imply degraded downstream performance in this regime.
Get next week’s papers.
The same picks and the same summaries, in your inbox. Free, and 176 issues deep.
Subscribe on Substack