Understanding and Mitigating Inference-Time Overreliance Using Agentic Memory
Luoxi Tang, Zhaohan Xi (Binghamton University) and colleagues study memory overreliance, where benign and correctly retrieved memories lower accuracy because they only partly match the current task, and propose MemTrim to control how memory evidence is reused.
Ask this paper
When it happens. Across benchmarks and memory systems (Mem0, A-Mem, MemGPT, Graphiti), memory helps when past experience applies and hurts when the current query shares much of the evidence but needs a different conclusion.
Partial overlap. Failures concentrate at intermediate query-memory overlap. Controlled experiments that vary overlap while fixing the rest of the memory reproduce the pattern, with the largest harm at middle overlap levels.
MemTrim. Indexes memory evidence at write time and, at read time, removes repeated or conflicting evidence, labels stored answers with the question they answered, and marks task changes.
Result. On task-divergent cases MemTrim recovers much of the accuracy lost under unmodified memory, while task-aligned cases stay at or above the original memory system. Removing entries or compressing memory reduces harm but also removes the benefit.
Practical value. No retraining, and it works with both embedding-based and structured memory stores.
Abstract
Agentic memory allows LLM agents to reuse past experience, yet retrieved memories can also distort inference even when they are benign, correctly stored, and appropriately retrieved. We study this failure mode, which we call memory over-reliance. Across benchmarks and memory architectures, we find that memory is useful when past experience transfers to the current task, but can become misleading when only part of the evidence transfers. Failures are strongest under partial query-memory overlap, a pattern further confirmed by controlled experiments thatvary the amount of overlapping evidence. Motivated by this finding, we propose MEMTRIM, a plug-and-play framework that indexes memory evidence at write time and controls its reuse at read time. MEMTRIM removes repeated or conflicting evidence while preserving useful memory-specific information, requires no retraining, and applies to both embedding-based and structured memory systems.Experiments show that MEMTRIM reduces memory overreliance while preserving the benefits of useful memory across models and memory settings.