Length Is Cheap, Competition Is Not: Attention Predicts Context Rot in Retrieval
Yinuo Liu (UC San Diego) shows that in retrieval tasks, context rot comes from statements that compete with the evidence, not from length, and that the attention mass left on the evidence predicts the failure.
Ask this paper
Length. On five open models (0.5B to 3B, four families), neutral filler costs nothing: accuracy stays at 0.97 to 1.00 up to 32K tokens on Qwen2.5-1.5B.
Competition. Statements formatted like the evidence, at the same positions, drop accuracy to 0.16 to 0.33. One rival value for the queried key multiplies the loss by 9 to 110.
Mechanism. Across 126,473 trials, attention mass on the evidence explains up to 58% of the variance in loss. Log length explains at most 8%.
Prior protocol. The masked-filler protocol behind the claim that length alone hurts fails a visible control: masking costs 45 points at 32K tokens while the same visible filler costs none.
Practice. For RAG and agent memory, deduplicating and resolving conflicting statements matters more than shortening context. All 126K trial-level measurements are released.
Abstract
Why do language models fail on long inputs? The common answer is length. We show that, for retrieval, length is cheap and competition is not. On five open models (0.5B to 3B parameters, four families), neutral filler costs nothing: accuracy stays at 0.97 to 1.00, up to 32K tokens on Qwen2.5-1.5B. Statements in the format of the evidence, placed at the same positions, drop it to 0.16 to 0.33. A single statement that gives the queried key a rival value multiplies the loss by 9 to 110, and every additional rival adds to it. Inside the model, one quantity tracks the failure: the attention mass left on the evidence. Across 126,473 trials it explains up to 58% of the variance in loss; log length explains at most 8%. The masked-filler protocol behind the claim that length alone hurts reproduces on MMLU (11 to 18 points lost at 30K tokens), but fails a visible control in retrieval: masking costs 45 accuracy points at 32K tokens, while the same filler left visible costs none. In retrieval, context rot is a competition for attention. We release the probe and all 126K trial-level measurements.