EvalMem: An Operation-Level Diagnostic Framework for Long-Term Memory Systems
Zeyu Liu, Jian Zhong and colleagues led by Nankai University (EMNLP 2026 Findings) introduce EvalMem, a framework that attributes long-term memory failures to encoding, retrieval or generation instead of reporting only end-to-end QA accuracy.
Ask this paper
Three examiners. For each query, one examiner checks whether the target fact was stored, one whether the native retriever returned usable evidence, and one whether the model answers correctly from oracle evidence. Their outputs combine into multi-label defect codes.
Store-level recall. A recall-first agentic RAG search that uses both the query and the source evidence raises recall of stored evidence on LoCoMo from 70.2% to 95.6%, which makes the encoding check reliable.
Retrieval fails most. Across seven memory systems on LoCoMo, LongMemEval-S and DynaMem-Bench, retrieval is the most frequently attributed failure layer. On default LoCoMo, retrieval defects reach 22.1% versus 7.7% for encoding and 6.5% for generation.
Fix guided by diagnosis. MemWiki, a search-friendly structure built from each system's memory export, raises mean accuracy by 2.5 points on LoCoMo and 2.3 on LongMemEval-S.
Abstract
Long-horizon interactions with LLM-based assistants require memory systems that preserve and update user states, preferences, and interaction histories. Existing evaluations report end-to-end QA accuracy and cannot determine whether errors arise from encoding, retrieval, or generation. We introduce EvalMem, an operation-level diagnostic framework with three parallel Examiners. For each query, the Encoding Examiner checks whether the target fact is stored, the Retrieval Examiner assesses whether the native retriever returns usable evidence, and the Generation Examiner tests whether the model can answer from oracle evidence. Their outputs form fine-grained multi-label defect codes. To improve store-level diagnosis, we adapt agentic RAG with a recall-first strategy that searches using both the query and source evidence, increasing recall of present evidence on LoCoMo from 70.2% to 95.6%. Evaluations of seven memory systems on LoCoMo, LongMemEval-S, and dynamic DynaMem-Bench identify retrieval as the most frequently attributed failure layer; in default LoCoMo, retrieval defects reach 22.1%, compared with 7.7% for encoding and 6.5% for generation. Guided by this diagnosis, MemWiki, a search-friendly auxiliary structure built from each system's memory export, improves mean accuracy by 2.5 and 2.3 percentage points on LoCoMo and LongMemEval-S, respectively.