What Eviction Destroys: A Restore-Counterfactual Audit of Forgetting in Agent Memory
Chen Shen at Megagon Labs introduces the restore counterfactual, a per-question intervention that puts the gold evidence back into a reader's context after eviction, which separates losses eviction destroyed permanently from losses retrieval merely failed to surface.
Ask this paper
What budget-accuracy curves cannot tell you: Existing frontiers quantify how much accuracy is lost at a given token budget but not whether the information is gone or merely unretrieved, which are different problems with different fixes.
The intervention: For each question the gold evidence is reinstated in the read-time context and the same reader is rerun. Combining the change in correctness with whether the evidence survived eviction classifies each oracle-answerable error as recoverable, irreversible, or residual.
Four policies at 80k tokens: Under top-k retrieval the irreversible share among errors corrected by restoration is 0.67 to 0.73 for FIFO, random and redundancy-aware eviction, against 0.60 for LLM-importance eviction.
At 8k tokens the policy stops mattering: The irreversible share reaches 1.00 for all four policies, so below a certain budget the choice of eviction rule is not what is limiting performance.
A methodological warning: Recoverable errors appear under top-k retrieval at 80k tokens but are absent under forced-gold injection by construction, so budget-accuracy results are not comparable unless the retrieval regime is reported.
Abstract
Agent memory systems must discard stored information when their history exceeds a fixed token budget. Existing budget-accuracy frontiers quantify the resulting loss in accuracy, but do not distinguish irreversible losses caused by eviction from recoverable retrieval failures. We introduce the restore counterfactual, a per-question paired intervention that reinstates the question's gold evidence in the read-time context and reruns the same reader. Combining the change in correctness with whether the evidence was retained after eviction classifies each oracle-answerable error as recoverable, irreversible, or residual; in the residual case, the answer remains incorrect after restoration. We evaluate FIFO, random, redundancy-aware, and LLM-importance eviction on LongMemEval-S at three budgets and under two retrieval regimes, using GPT-4o-mini as the primary reader and judge and GPT-5.4-mini as a robustness reader. Under top-k retrieval at an 80k-token budget, the irreversible share among errors corrected by restoration is 0.67-0.73 for FIFO, random, and redundancy-aware eviction, compared with 0.60 for LLM-importance. At 8k tokens, it reaches 1.00 for all four policies. Recoverable errors occur under top-k retrieval at 80k tokens but are absent under forced-gold injection by construction, so budget-accuracy results are not directly comparable unless the retrieval regime is reported. An exploratory matched-accuracy analysis detects no difference in irreversible rate among accuracy-matched policy pairs at a resolution of 1.2-6 percentage points. The same analysis detects the deliberately destructive control. To our knowledge, this is the first per-item, per-question restore-counterfactual audit of eviction for external agent-memory stores on a standard conversational benchmark.