Memory Tax: Dissecting the Net Value of Agent Memory
Xuan He, Sining Zhoubian, ChengXiang Zhai, Jiawei Han and colleagues (University of Illinois Urbana-Champaign, Tsinghua University, Renmin University, University of Waterloo) measure the net value of agent memory systems against naive RAG over raw history, and find it is often negative.
Ask this paper
Main finding. Across LongMemEval-s, PersonaMem-v2 and MINT-Eval and several model backbones, many dedicated memory systems underperform naive RAG over the raw history, despite substantially higher costs.
Benefit and tax. Per-query comparison with RAG splits net value into memory benefit (queries memory solves that RAG misses) and memory tax (queries RAG solves that memory loses).
Where the tax comes from. Memory processing can drop or distort information (faithfulness) and make preserved information harder to retrieve (accessibility). It also makes results more sensitive to the choice of backbone model.
Fix. A proof of concept that also retrieves from the source history and reranks it jointly with memory reduces the tax, which supports keeping a path back to raw history.
Abstract
Memory is increasingly used in long-horizon agents, yet the net value of building memory over direct retrieval from raw history remains largely unclear. Through a large-scale evaluation across multiple benchmarks and model backbones, we find many dedicated memory systems underperform naive RAG in many cases, indicating a negative net value. Taking RAG as a reference, we dissect the net value of memory into the gains and losses over direct retrieval, termed memory benefit and memory tax. These gains and losses arise because memory process changes how the raw information is organized and accessed. We view such changes as the double-edged deviations from the raw history. They are originally designed and introduced to benefit the task, for example by reorganizing facts or distilling preferences, but some may also remove or obscure evidence, giving rise to memory tax. Since memory benefit reflects the expected gains of its design, we then focus more on the overlooked memory tax, study how it arises from the system and evaluation side, and generalize the concept to more application scenarios of memory. We hope the notion of memory tax can provide a new lens and shift more attention from standalone performance to the net value of memory, informing more reliable memory designs and evaluations. We will open-source the code and results to the community. Project Page: https://hexuan21.github.io/MemoryTax/