MemoryArena
Free while signed in. Answers cite the passages they came from.
MemoryArena introduces a benchmark for evaluating how agents utilize memory across multiple interconnected sessions. The key finding is that scoring well on memory recall does not mean an agent can actually use that memory to take correct actions across sessions. Models with near-saturated performance on existing benchmarks like LoCoMo perform poorly in agentic multi-session settings. - **Agentic memory evaluation:** Unlike standard memory benchmarks that test recall in isolation, MemoryArena evaluates whether agents can retrieve and apply relevant past experience to make correct decisions in new contexts. This exposes a gap between retrieval accuracy and actionable memory use. - **Interdependent multi-session tasks:** The benchmark spans web navigation, constrained planning, information retrieval, and logical reasoning, where decisions in one session depend on information gathered in previous sessions. This reflects real-world agent deployments where sessions are not independent. - **Exposing evaluation blind spots:** Agents achieving near-perfect scores on LoCoMo and other long-context benchmarks show significant performance drops on MemoryArena. This suggests current evaluations overestimate agent memory capabilities by testing retrieval without testing downstream decision quality. - **Practical implications:** For developers building persistent agents, MemoryArena provides a more realistic assessment of whether memory systems actually improve task completion rather than just information access.
Agentic memory evaluation: Unlike standard memory benchmarks that test recall in isolation, MemoryArena evaluates whether agents can retrieve and apply relevant past experience to make correct decisions in new contexts. This exposes a gap between retrieval accuracy and actionable memory use.
Interdependent multi-session tasks: The benchmark spans web navigation, constrained planning, information retrieval, and logical reasoning, where decisions in one session depend on information gathered in previous sessions. This reflects real-world agent deployments where sessions are not independent.
Exposing evaluation blind spots: Agents achieving near-perfect scores on LoCoMo and other long-context benchmarks show significant performance drops on MemoryArena. This suggests current evaluations overestimate agent memory capabilities by testing retrieval without testing downstream decision quality.
Practical implications: For developers building persistent agents, MemoryArena provides a more realistic assessment of whether memory systems actually improve task completion rather than just information access.
Abstract
Existing evaluations of agents with memory typically assess memorization and action in isolation. One class of benchmarks evaluates memorization by testing recall of past conversations or text but fails to capture how memory is used to guide future decisions. Another class focuses on agents acting in single-session tasks without the need for long-term memory. However, in realistic settings, memorization and action are tightly coupled: agents acquire memory while interacting with the environment, and subsequently rely on that memory to solve future tasks. To capture this setting, we introduce MemoryArena, a unified evaluation gym for benchmarking agent memory in multi-session Memory-Agent-Environment loops. The benchmark consists of human-crafted agentic tasks with explicitly interdependent subtasks, where agents must learn from earlier actions and feedback by distilling experiences into memory, and subsequently use that memory to guide later actions to solve the overall task. MemoryArena supports evaluation across web navigation, preference-constrained planning, progressive information search, and sequential formal reasoning, and reveals that agents with near-saturated performance on existing long-context memory benchmarks like LoCoMo perform poorly in our agentic setting, exposing a gap in current evaluations for agents with memory.
Get next week’s papers.
The same picks and the same summaries, in your inbox. Free, and 176 issues deep.
Subscribe on Substack