Context Rot

Simple tasks reveal degradation: Even basic benchmarks like semantic variants of Needle-in-a-Haystack, repeated word copying, or long QA logs (LongMemEval) expose accuracy drops as context length increases. The decline is more dramatic for semantically ambiguous inputs or outputs that scale with length.

Distractors and structure matter: The presence of plausible distractors significantly reduces accuracy, with different distractors affecting models to varying degrees. Surprisingly, models often perform better on shuffled (structureless) haystacks than logically coherent ones, suggesting that attention is disrupted by narrative flow.

Similarity and position effects: Lower semantic similarity between a query and its answer leads to faster degradation with context length. Models also show a preference for needles appearing early in context and struggle with information retrieval when the needle blends into the haystack thematically.

Repetition and refusal behaviors: In repeated-word tasks, autoregressive degradation appears in long outputs, with models hallucinating, refusing to generate, or inserting random tokens. Performance varies even within model families, with conservative models like Claude Opus 4 often abstaining, while GPT-4.1 more frequently hallucinates.