Experience-Handoff Reinforcement Learning for Recursive Self-Improvement
Doyoung Kim, Byeongguk Jeon, Kailin Zhang and Sherry Yang at NYU and KAIST AI propose ACE-RL, which trains one model to act, reflect and curate playbooks inside the Agentic Context Engineering harness, scoring each handed-off artifact by how much it helps later rollouts.
Ask this paper
Motivation. Outcome-only RL reduces a trial to one scalar and discards error logs and tool outputs. Harnesses that hand experience forward keep that information, but the model writing the handoff is not trained.
Method. Within a fixed ACE scaffold, the actor, insight reflector and playbook curator roles are co-optimized. Each reflection or playbook edit is credited by its downstream effect on future actor rollouts, rather than every output in a chain sharing the final reward.
Results. At equal training compute on Mastermind, MBPP, AppWorld and MiniSWEBench, ACE-RL reaches higher held-out reward than actor-only GRPO without the harness and end-to-end training through the harness with a learned critic.
RSI@k. On AppWorld with Qwen3-4B-Instruct, ACE-RL solves more held-out tasks across eight consecutive attempts than GRPO, because later attempts inherit better playbooks.
Takeaway. Agent training should optimize both the current actions and the experience the model writes for its future rollouts.
Abstract
It is common to train Large Language Models (LLMs) for agentic tasks with Reinforcement Learning (RL) from scalar rewards. However, this throws away the fine-grained information about the environment that the agent experienced during the trial (e.g., error logs). To utilize this rich information, recent works wrap the agent in a harness that refines its own inputs through reflection or compaction and hands past experience to future rollouts, but the construction of the handed-off artifacts (e.g., a playbook) is itself not learned. We want the best of both worlds: gradient updates from a scalar reward, while also utilizing environment information to hand off experience. As a concrete testbed, we fix the Agentic Context Engineering (\ace{}) harness and train one model to act, reflect, and curate task-local playbooks. Our method, \acerl{}, evaluates candidate experience-handoff artifacts by their downstream effect on future actor rollouts, and co-optimizes the role-specific behaviors (action generator, insight reflector, and playbook curator) under a fixed scaffold. Across agentic benchmarks, \acerl{} achieves higher held-out performance than baseline methods. The results suggest that agent training should optimize not only the actions in the current rollout, but also the model-written experience that future rollouts inherit.