Enhancing Long-Term Memory in LLMs
Researchers from University of Alberta and UMass Amherst introduce BEAM, a new benchmark for evaluating long-term memory in LLMs with conversations up to 10M tokens, and LIGHT, a framework that enhances memory performance through three complementary systems. - **Novel benchmark design**: 100 diverse conversations (100K-10M tokens) with 2,000 validated questions testing 10 memory abilities including contradiction resolution, event ordering, and instruction following. - **Advanced generation framework**: Automatically creates coherent narratives across 19 domains using conversation plans, user profiles, relationship graphs, and bidirectional dialogue dynamics. - **Cognitive-inspired architecture**: LIGHT integrates episodic memory (retrieval-based), working memory (recent turns), and scratchpad (salient facts) mimicking human memory systems. - **Strong empirical results**: 3.5-12.69% average improvement over baselines, with largest gains in summarization (+160.6%), multi-hop reasoning (+27.2%), and preference following (+76.5%). - **Scalability at extreme lengths**: At 10M tokens, LIGHT shows +155.7% (Llama-4-Maverick) and +107.3% (GPT-4.1-nano) improvements where no baseline supports full context. - **Ablation insights**: At 10M tokens, removing retrieval (-8.5%), scratchpad (-3.7%), working memory (-5.7%), or noise filtering (-8.3%) significantly degrades performance.
Ask this paper
Novel benchmark design: 100 diverse conversations (100K-10M tokens) with 2,000 validated questions testing 10 memory abilities, including contradiction resolution, event ordering, and instruction following.
Advanced generation framework: Automatically creates coherent narratives across 19 domains using conversation plans, user profiles, relationship graphs, and bidirectional dialogue dynamics.
Cognitive-inspired architecture: LIGHT integrates episodic memory (retrieval-based), working memory (recent turns), and scratchpad (salient facts), mimicking human memory systems.
Strong empirical results: 3.5-12.69% average improvement over baselines, with the largest gains in summarization (+160.6%), multi-hop reasoning (+27.2%), and preference following (+76.5%).
Scalability at extreme lengths: At 10M tokens, LIGHT shows +155.7% (Llama-4-Maverick) and +107.3% (GPT-4.1-nano) improvements, where no baseline supports full context.
Ablation insights: At 10M tokens, removing retrieval (-8.5%), scratchpad (-3.7%), working memory (-5.7%), or noise filtering (-8.3%) significantly degrades performance.
Abstract
Evaluating the abilities of large language models (LLMs) for tasks that require long-term memory and thus long-context reasoning, for example in conversational settings, is hampered by the existing benchmarks, which often lack narrative coherence, cover narrow domains, and only test simple recall-oriented tasks. This paper introduces a comprehensive solution to these challenges. First, we present a novel framework for automatically generating long (up to 10M tokens), coherent, and topically diverse conversations, accompanied by probing questions targeting a wide range of memory abilities. From this, we construct BEAM, a new benchmark comprising 100 conversations and 2,000 validated questions. Second, to enhance model performance, we propose LIGHT-a framework inspired by human cognition that equips LLMs with three complementary memory systems: a long-term episodic memory, a short-term working memory, and a scratchpad for accumulating salient facts. Our experiments on BEAM reveal that even LLMs with 1M token context windows (with and without retrieval-augmentation) struggle as dialogues lengthen. In contrast, LIGHT consistently improves performance across various models, achieving an average improvement of 3.5%-12.69% over the strongest baselines, depending on the backbone LLM. An ablation study further confirms the contribution of each memory component.