Delivered, Not Used: Evaluating Agent Memory at Its Point of Use
Changran Xu, Zeju Li and Qiang Xu at The Chinese University of Hong Kong introduce MemUse, an evaluation that judges agent memory at the moment each memory is delivered and follows what the agent does with it afterward.
Ask this paper
Method. Each memory delivery is compared with what the agent could already see. An agentic judge rates relevance, accuracy, novelty, influence and observed effect, programmatic checks enforce that cited evidence was read in full, and fixed rules turn the ratings into verdicts linked to memory histories and token cost.
Main result. On 8,000 SWE-bench Verified episodes from three memory systems (MemGovern, Mem0, TencentDB), resolve rates stay within 0.4 points of a no-memory baseline. MemUse shows that most delivered entries were irrelevant, most relevant entries went unused, and in two systems most deliveries repeated content the agent could still see.
Hidden effects. 23.2% of episodes with beneficial memory use still failed, and 5 of 16 episodes with harmful memory use still succeeded, so task outcomes alone hide both.
Design guidance. Full cards, task recall and full skills supply most beneficial entries at small token shares, while previews and indexes mainly help selection. Filtering, deduplication and timing of delivery are the main levers.
Abstract
Task success, retrieval quality, and cost do not show whether an LLM agent used a delivered memory or what followed from that use. We introduce MemUse, which evaluates memory at its point of use in recorded trajectories: each memory delivery is compared with what the agent could already see and followed through the agent's later actions and their consequences. An agentic judge rates each delivered entry's relevance, accuracy, novelty, influence, and observed effect; programmatic checks enforce complete reading of cited evidence, and fixed rules aggregate the ratings into verdicts linked to memory histories and delivery cost. On 8,000 SWE-bench Verified episodes from three memory systems, resolve rates stay within 0.4 percentage points of a no-memory baseline, yet MemUse shows that most delivered entries were irrelevant, most relevant entries went unused, and in two systems most deliveries restated content the agent could still see. It also surfaces effects that outcomes hide: 23.2\% of episodes with beneficial memory use still failed, and 5 of 16 with harmful use still succeeded.