Route to Remember: Adaptive Memory Selection for LLM Agents
Zeyu Fang, Chenlei Guo and colleagues at George Washington University and Amazon store user facts in model weights with on-policy self-distillation (OPSD) and add a verify-then-regenerate router that falls back to text memory when needed.
Ask this paper
Method. OPSD trains a context-free student from a teacher that is the same model conditioned on the user's memory, so the student answers memory questions with no retrieved context.
Recall. On eight HaluMem-Long users, OPSD reaches 59.9% QA accuracy versus 47.8% for MemOS-style retrieval, 51.9% for extracted context and 51.3% for raw dialogue. Plain SFT on the same data reaches 36.9%, below every text baseline.
Scale. The advantage over retrieval holds for all eight users and at 4B, 9B and 27B.
Failure modes. Weight memory handles recent facts well but forgets as facts age and cannot tell when a fact was never recorded. Refusal behavior depends on the share of refusal samples in training.
Router. Verify-then-regenerate adds 6.2 points over OPSD and improves five of six question categories. Query-side signals such as similarity or confidence barely predict when weight memory is right.
Abstract
The memory of production agents is increasingly becoming a bottleneck and is built almost exclusively from text: key points are extracted from out-of-limit context and retrieved into context at query time. Therefore, every new query incurs an additional cost that grows with the accumulated history. Weight memory is a complementary alternative that internalizes knowledge into model parameters and answers with no context at all, and has recently proven effective in self-evolving agents for internalizing skills and behaviors. However, its application in episodic and semantic memory, such as user-specific facts, remains comparatively unexplored, and a gap remains between earlier attempts at weight memory and text-based systems regarding recall performance. We study how to make weight memory competitive as a long-term factual store by applying on-policy self-distillation (OPSD). It trains a context-free student from a teacher, which is the same model conditioned on the user's memory, together with a verify-then-regenerate router that lets weight and text memory each cover the queries they handle best. On the HaluMem benchmark, OPSD matches or surpasses strong text-memory baselines, including full-context prompting and retrieval systems, while reading fewer tokens per query. We further conduct an extensive study that characterizes the behavior of weight memory and analyzes how different training recipes affect its performance. As agent harnesses move toward persistent, multi-session deployments, these results position weight memory as a practical, low-cost component of agent memory and provide insights into feasible training and evaluation protocols for combining it with retrieval.