Environmental Feedback Modeling Matters: Rethinking Feedback Treatment in Agentic Hindsight Self-Distillation
Hangxi Guo, Mengnan Du and colleagues (The Chinese University of Hong Kong, Shenzhen, with Shanghai AI Laboratory, USTC and ICT CAS) present SELF, which trains a language agent both to predict environment feedback and to distill from a feedback-conditioned copy of itself, for settings where rewards are unavailable.
Ask this paper
Problem. Recent hindsight self-distillation conditions the teacher on environment feedback, but the authors find that this alone uses the feedback poorly. Giving SDPO the full feedback lowers its success from 30.4% to 22.5% with Qwen3-4B-Instruct.
Method. SELF adds an objective that predicts the environment's response to each action, trained jointly with distillation from the feedback-conditioned self-teacher.
Results. With Qwen3-8B, SELF beats SDPO and GRPO by 6.4 and 4.1 points on tau-bench success and by 10.71 and 3.57 points on AppWorld task goal completion. With Qwen3-4B-Instruct it reaches 38.7% against 19.2% to 27.4% for Mem0.
The two objectives help each other. Adding feedback modeling lifts teacher success from 30.1% to 46.7% and student success from 29.4% to 38.7%. Self-distillation raises feedback prediction accuracy on Retail from 37% to 73%.
No general regression. MMLU, MMLU-Pro and IFEval move by at most 0.11 points after training.
Abstract
Reinforcement learning is commonly used to train language agents in interactive environments, but cannot be directly applied when rewards are unavailable. Recent methods use environmental feedback as privileged context for hindsight self-distillation, but our analysis suggests that simply conditioning the teacher on feedback is insufficient, motivating us to rethink how environmental feedback is used in agentic self-distillation. Given that environmental feedback contains rich supervision for modeling how the environment responds to agent actions, we introduce \textit{agentic SElf-distilLation with environmental Feedback modeling} (SELF), a framework that jointly optimizes environmental feedback modeling and hindsight self-distillation. SELF learns to predict environmental responses while distilling guidance from a feedback-conditioned self-teacher into the policy. Our analysis reveals a mutually reinforcing mechanism: environmental feedback modeling strengthens hindsight supervision and policy learning, while self-distillation enhances the model's ability to model environmental feedback. With Qwen3-8B, SELF outperforms SDPO and GRPO by 6.4 and 4.1 percentage points in $τ$-bench success rate, and by 10.71 and 3.57 percentage points in AppWorld task goal completion, respectively. These results show that SELF uses environmental feedback more effectively within agentic self-distillation, improving agent capabilities.