Self-Evolving Agents through Privileged Retrospective Distillation
Yuhang Zhou, Zhuokai Zhao and colleagues at Meta AI propose Retrospective Self-Distillation (RSD), which turns an agent's own unlabeled trajectories into token-level supervision by having the same model re-score its actions with privileged hindsight context.
Ask this paper
Same model, two roles. As teacher, the model sees cross-task precedents, stored lessons, turn-level reflection and the completed trajectory, and re-scores the actions it took as student. That teacher distribution is distilled back through on-policy self-distillation.
No labels in the core update. The policy update uses no ground-truth trajectories and no verifier-derived rewards.
Multiple rounds. The experience bank grows each round and the retrospective context is rebuilt, so the model, its data and its supervision change together.
Results. On ALFWorld, ScienceWorld and SearchQA with Qwen3-8B and Qwen3-14B, RSD improves performance by up to 59.7% relative to the base model. Letting a stronger frontier model propose constrained edits to how the privileged context is built adds up to 44.7% relative over a fixed context policy.
Abstract
Self-evolving agents aim to improve from their own experience rather than remain fixed after deployment, but naturally collected trajectories often lack ground-truth solutions or verifier-derived rewards. We introduce Retrospective Self-Distillation (RSD), a multi-round framework that turns such experience into dense supervision using privileged retrospective context. For each on-policy trajectory, the same underlying model acts as both student and teacher: the teacher re-scores the student's realized actions while conditioning on cross-task precedents and lessons, together with turn-level reflection and future hindsight from the completed trajectory. The resulting teacher distribution is distilled back into the policy through on-policy self-distillation. Across rounds, the experience bank grows and the retrospective context is refreshed, allowing the model, its data, and its supervision to co-evolve. We further introduce frontier-guided context-policy evolution, where a stronger frontier model proposes constrained edits to how privileged context is constructed. Across ALFWorld, ScienceWorld, and SearchQA with Qwen3-8B and Qwen3-14B, RSD improves performance by up to 59.7% relative to the base model, while frontier-guided context-policy evolution adds up to a further 44.7% relative improvement over a fixed context policy. These results show that naturally accumulated experience can be converted into persistent model capability without ground-truth trajectories or verifier-derived rewards for the core policy update.