Persistent Memory in Multi-Agent LLM Inference: What It Costs, What It Buys, and When You Can Tell
Hochan Son, Guang Cheng (UCLA) and colleagues from Wisconsin and HCLTech measure what a persistent memory tier adds to a chain-of-agents long-context system, and find that decomposition cuts peak KV cache while the persistent tier adds cost with no detectable accuracy change.
Ask this paper
Decomposition bounds the KV working set. Splitting long-context inference across cooperating agents brings the peak KV cache per query to 14.3 MiB, against 35.5 MiB for single-pass and 35.3 MiB for a retrieval-matched RAG baseline, with latency of 29 s against 67 s and 52 s.
RAG does not shrink the largest call. The RAG baseline still concatenates its passages into one generation call, so filtering evidence changes which tokens enter the call but not the peak memory.
Persistent recall has a null effect. Across eight controlled dataset pairs at n=100 per arm, the trace-recall tier costs +0.368 MiB of peak cache and changes accuracy by +0.015 (95% CI -0.011 to +0.046).
The null is structural. Single-question benchmarks give each item its own evidence and score it independently, and correct ablations must reset stored traces between conditions, so recall has nothing useful to retrieve.
Four measurement corrections. Reaching the null took four fixes, three of which had inflated the apparent memory benefit, and none showed up in the results table. The paper lists the conditions an agent-memory ablation must meet.
Abstract
Decomposing long-context inference across cooperating agents bounds the active KV cache per call rather than total evidence, which matters when KV-cache memory binds. Many such systems add a persistent tier storing and recalling reasoning traces, usually validated by an ablation reporting an accuracy gain. We measure both on one three-tier agent architecture. Decomposition delivers: peak KV working set of 14.3 MiB per query against 35.5 and 35.3 MiB for single-pass and retrieval-augmented baselines. The persistent tier does not: across eight controlled dataset pairs at n=100 per arm it costs +0.368 MiB [+0.167, +0.590] of peak cache and produces no detectable accuracy change (+0.015, 95% CI [-0.011, +0.046]). We argue the null is structural: single-question benchmarks supply each item with its own evidence and score it independently, and correctness requires resetting stored traces between conditions, so recall has nothing informative to retrieve. Reaching it took four measurement corrections -- three inflating the apparent benefit, the fourth making an effect that size look resolvable -- none visible in the results table. We give the conditions an agent-memory ablation must satisfy and detection procedures that need no knowledge of the specific defect.