Why Bother Me Again? Benchmarking Personal Agent Memory for Interaction Preferences
Jialin Li, Hanan Salam and colleagues (New York University Abu Dhabi) build MemPABench, an interactive environment that tests whether personal agents remember how a user wants to be assisted, not only what the user said.
Ask this paper
Environment. Four grounded simulated users hold 14 interaction preferences across work and personal contexts, with feedback over 99 sessions spanning four months and five event-driven preference changes per user.
Decisions tested. Agents must decide when to clarify, confirm or act autonomously, and are scored on preference selection, delivered behavior, and sensitivity to context and change.
Result. Retrieval-based memory (Mem0, MemOS, simple RAG) does not outperform No Memory. File-based memory improves accuracy but applies preferences in contexts where they do not belong.
Default habits. Even with the correct preference in context, agents struggle to override their default interaction behavior.
Abstract
Personal intelligence systems are taking on increasingly complex everyday tasks, yet repeated questions, unnecessary confirmations, and unwanted outreach can turn capable assistance into interaction burden. Reducing this burden requires agents to remember how users want to be assisted. However, existing memory benchmarks focus on factual recall and personalized answers and rarely test how agents learn and act on interaction preferences. We introduce MemPABench, an interactive environment for on-policy evaluation of memory in personal agents. MemPABench poses long-horizon tasks requiring agents to decide when to clarify, confirm, or act autonomously. Four grounded simulated users hold 14 preferences across work and personal contexts. Their feedback spans 99 sessions over four months, with five event-driven changes per user testing memory updates. We evaluate preference selection, delivered behavior, and sensitivity to context and preference change. Evaluating four memory systems across four LLM backbones, we find that current memory brings limited gains in preference selection. Retrieval-based systems (i.e. Mem0, MemOS, simple RAG) do not outperform No Memory, while File-based memory improves accuracy but applies preferences beyond relevant contexts. Our analysis shows that current memory struggles to retrieve interaction preferences and adapt to preference changes. Even with the correct preference available, overriding the agent's default interaction habits remains challenging. The environment and dataset are available at https://mempabench.com.