One Assistant, Many Memories: Benchmarking Real-World Group Assistance When Recall Is Not Enough
Zhanghao Hu, Lin Gui and colleagues at Tencent's Yuanbao team, King's College London and partner universities release GroupAssistBench, the first memory benchmark built from real group conversations with a deployed assistant, and show that memory systems recall well but use memories poorly.
Ask this paper
Data. 80,299 messages from 50 real groups, collected at a cost of over $53K. Requests are verified to require group history, and each needed memory is anchored to its source messages.
Two levels. L1 tests recall with reference answers. L2 tests whether the assistant uses memories to serve requests that have no fixed answer, scored with per-request rubrics derived from real user feedback.
Memory types. Each request needs a single memory, memories distributed across members, or conflicting memories (M1 to M3).
Finding. The strongest memory system rivals full context on recall but falls far behind it on use, and the gap widens for distributed and conflicting memories. Mem0, A-MEM, MemoryOS, EverMemOS, GraphRAG and RAPTOR were among the systems tested.
Fixes. Relational Evidence Refinement and Assistance Constraint Binding, which relate retrieved memories and turn remembered facts into response constraints, improved all three memory systems they were applied to.
Abstract
The value of memory lies not only in recall but in using it to provide appropriate assistance. Recent benchmarks test memory use in personalisation and agentic tasks, but seldom evaluate assistance without fixed answers or integration across multiple participants. Group chats make both challenges more acute: an assistant must integrate information from multiple members, track revisions and conflicts over time, and produce one response for the group. Yet existing group memory benchmarks rely on synthetic conversations and inherit constrained evaluation from general memory benchmarks, focusing on recall against fixed answers or decisions over a closed set. We introduce GroupAssistBench, the first memory benchmark from real-world group conversations with a deployed group assistant, covering 80,299 messages in 50 groups and costing over $53K. Over the same histories, GroupAssistBench evaluates memory recall (L1) with reference answers and memory use (L2) with service requests without fixed answers across single, distributed, and conflicting memories (M1--M3). Requests are verified to require group history, memories are anchored to their sources, and L2 uses tailored rubrics with penalties derived from real user feedback. Across evaluations of frontier models with full context, retrieval methods, and memory systems, the strongest memory system rivals full context on recall but falls far behind on use, especially for distributed and conflicting memories. Together, two additions that relate retrieved memories and bind remembered facts as response constraints bring consistent gains, showing that group memory systems must improve use, not recall alone.