When Users Don't Ask: Benchmarking Context-Driven Memory Retrieval in Conversational Agents
Wen-Yu Chang and Yun-Nung Chen build LOCOMO-CONV, a conversational memory benchmark that replaces QA-style probing with in-situ dialog usage, and find retrieval gaps that QA benchmarks simply do not see.
Ask this paper
Four query styles: dialog, implicit, counterfactual and composed. The implicit and composed styles are where the gaps open, and both are what real usage actually looks like.
Query rewriting only half-fixes it: multi-facet rewriting narrows the gap for raw-turn memory but not for abstractive memory, which is a useful discriminator between memory architectures.
Strong retrieval does not buy response quality: the two metrics come apart, so leaderboard recall is not a proxy for a better assistant.
Silent grounding: on implicit queries, memory improves contextual grounding without ever surfacing the gold fact, a failure mode that gold-fact scoring cannot detect.
Released supportive_memory annotations capture conversationally useful context beyond the original gold evidence, which is the reusable artifact here.
Abstract
Large language models (LLMs) are increas- ingly deployed as long-horizon conversational agents, motivating growing interest in mem- ory systems. However, existing benchmarks primarily evaluate memory through QA-style probing rather than in-situ conversational usage. We introduce LOCOMO-CONV, a conversa- tional memory benchmark derived from Lo- CoMo with four query styles: dialog, implicit, counterfactual, and composed. Across five rep- resentative memory systems, we evaluate both retrieval recall and end-to-end response qual- ity. Our experiments show that conversational framing exposes substantial retrieval gaps over- looked by QA benchmarks, especially on im- plicit and composed queries, which multi-facet query rewriting narrows for raw-turn mem- ory but not abstractive memory. We further find that strong retrieval does not fully trans- late into response quality, and that implicit queries exhibit silent grounding, where mem- ory improves contextual grounding without ex- plicitly surfacing the gold fact. These results point to reasoning-based memory elaboration as a promising direction, and we release aux- iliary supportive_memory annotations captur- ing conversationally useful context beyond the original gold evidence.