MemComposer: Learning to Assemble Memories for LLM Agents via Reinforcement Learning
Yixuan Xie, Xueqiang Xu, Jiawei Han and colleagues (UIUC, with Amazon co-authors) train MemComposer, a model that decides which retrieved experiences to give an agent, how many, and in what order, using task success from full agent interactions as the reward.
Ask this paper
Problem. Semantic similarity does not guarantee that a retrieved memory helps. Downstream performance depends on which memories are selected and how they are presented.
Composer. MemComposer reads the current query together with the retrieved candidates and outputs an assembled memory context: a subset, its size and its order.
ExPO training. Experience Policy Optimization is a GRPO-based method that runs complete agent episodes with each assembled context and uses task success as the reward for the assembly policy.
Results. Across three tau-squared-bench domains with two frozen executors, MemComposer beats the strongest baseline for each executor and cuts memory-token usage by 16.9% and 4.9%.
Transfer. The policy transfers to unseen domains, executors and memory banks, and benefits from memories written by stronger models without retraining.
Abstract
Agent memory enables large language model (LLM) agents to reuse past experiences without updating their parameters. However, semantic similarity alone does not guarantee memory usefulness: downstream performance depends on how memories are selected and presented to the agent. We propose MemComposer, a memory composer that jointly examines the current query and retrieved candidates to decide which experiences to include, how many to select, and how to order them. We train the composer using Experience Policy Optimization (ExPO), a GRPO-based framework that evaluates assembled memory contexts through complete agent interactions and uses task-success feedback to optimize the assembly policy. Across three $τ^2$-bench domains with two frozen executors, MemComposer outperforms the strongest baseline for each executor while reducing average memory-token usage by 16.9% and 4.9%, respectively. The learned policy also transfers to unseen domains, executors, and memory banks, and benefits from memories generated by stronger models without further training. These results demonstrate the value of learning query-specific memory assembly using downstream task feedback.