Memory-Aware Reward Modeling: Learning Human Preferences for LLM Memory Use
Yijiang River Dong, Jiaxin Pei and colleagues (Cambridge, Stanford, TUM, Harvard, UT Austin) test whether users actually prefer memory-conditioned assistant responses, using their own ChatGPT queries and memories.
Ask this paper
Data. About 1,000 pairwise judgments from 87 participants on their own queries and personal memories.
Finding. Memory-conditioned responses are not consistently preferred, and in some cases users strongly prefer responses generated without memory.
Evaluator gap. Automatic evaluators agree poorly with users and favor memory-conditioned responses much more often than users do.
Model. A memory-aware reward model trained on user judgments reaches 72.8% preference prediction accuracy, above existing baselines.
Abstract
Memory enables language model assistants to maintain continuity across sessions and generate personalized responses, but knowing more about a user does not necessarily produce responses they prefer. Existing evaluations assess memory capabilities without directly establishing whether memory use benefits the user. To study this gap, we collect approximately 1,000 pairwise judgments from 87 participants using their own ChatGPT queries and personal memories. We find that memory-conditioned responses are not consistently preferred over responses generated without memory and in some cases, users strongly prefer the latter. Automatic evaluators also show limited agreement with users and favor memory-conditioned responses substantially more often than users do. Motivated by these gaps, we introduce a memory-aware reward model that learns from user judgments, achieving 72.8\% preference prediction accuracy and outperforming the existing baselines. Our findings motivate evaluation and optimization objectives that align memory use with the preferences of the people whose information is being used.