Elastic-Cache
A training-free, architecture-agnostic way to make diffusion LLM decoding fast by updating KV caches only when and where it matters. Instead of recomputing QKV for all tokens at every denoising step, Elastic-Cache watches attention drift on the most-attended tokens and refreshes only deeper layers while reusing shallow and off-window caches. Results: large speedups with minimal or no accuracy loss across math, code, and multimodal tasks.
Ask this paper
Core idea: Sliding-window decoding keeps only nearby MASK tokens “live” and block-caches distant MASKs as a length prior. An attention-aware drift test measures cosine similarity changes of the previous step’s most-attended tokens; if similarity drops below a threshold γ at layer ℓ, recomputation starts from ℓ+1 to L. Shallow layers reuse caches; deep layers refresh.
Why this works: KV drift is small across most steps and grows with depth, so refreshing all layers is wasteful. The most-attended token shows the least KV change, giving a conservative lower bound to trigger refreshes. Visualizations support: distant MASKs have little influence; KV and attention changes align; most-attended tokens drift least.
Algorithm knobs for practitioners: Threshold γ controls the speed-accuracy tradeoff; lower γ updates less and runs faster. Window size β trades per-step compute for fewer steps. Works with confidence-aware parallel decoding (ϵ) and shows low update frequency even at higher γ. Defaults used: γ 0.9, ϵ 0.9, typical β 16–32.
Results that matter: On LLaDA and LLaDA-1.5: up to 45.1× throughput on GSM8K-512 with equal accuracy, 8.7× on GSM8K-256, and 4.8–5.0× on HumanEval with accuracy maintained or improved vs baselines. On LLaDA-V, throughput rises while preserving MathVerse accuracy. Elastic-Cache consistently beats Fast-dLLM in tokens/sec at comparable or better accuracy, and its throughput scales favorably with longer generations.
Deployment notes: No training or architecture changes required. Compatible with existing confidence-based and interval policies. Includes a practical batch implementation that concatenates variable-length sequences to preserve parallelism. Ethical and reproducibility details plus code plans included.