Block Diffusion

Combining AR and diffusion – Standard diffusion language models are fixed-length and slow to generate, while AR models generate token-by-token. Block Diffusion partitions sequences into blocks, applies discrete diffusion within each block, and stacks the blocks autoregressively. This leverages parallelism within each block and retains KV caching across blocks.

Efficient, flexible-length generation – BD3-LMs break free from fixed-size diffusion constraints. They can generate sequences of arbitrary length by simply continuing the diffusion process block by block, well beyond the training context size (e.g. thousands of tokens).

High likelihood and faster sampling – Prior diffusion LMs often lag behind AR in perplexity and need many denoising steps. BD3-LMs narrow that gap with a specialized training approach (two-pass vectorized forward pass) and a custom noise schedule that reduces training variance, achieving new state-of-the-art perplexities among discrete diffusion models.

Block-size tradeoffs – Smaller block sizes (e.g. 4 tokens) enable more parallel sampling but require more block steps. Larger block sizes (e.g. 16 tokens) reduce total steps but yield slightly higher variance. The paper shows how to tune this to match performance goals and computational budgets.

Open-source and generalizable – The authors provide code, model weights, and a blog post with examples. Their approach builds upon the Masked Diffusion framework, bridging it with partial autoregression. Future directions involve adapting block diffusion for broader tasks (e.g., chatbots, code generation) with flexible controllability.