Reinforcement Learning via Self-Distillation
Free while signed in. Answers cite the passages they came from.
This paper introduces Self-Distillation Policy Optimization (SDPO), an on-policy RL algorithm that converts rich textual feedback from verifiable environments into dense credit assignment without requiring an external teacher model. SDPO uses the current model conditioned on feedback as a "self-teacher" to retrospectively identify mistakes in its own rollouts, substantially outperforming GRPO across scientific reasoning, tool use, and competitive programming. - **Self-teacher for dense credit assignment:** Instead of learning from sparse scalar rewards like GRPO, SDPO re-evaluates the model's original attempt after conditioning on environment feedback (runtime errors, failed tests, or successful rollouts). This produces logit-level advantages at every token position, compared to GRPO's constant per-rollout advantages. The approach requires only minor changes to standard RLVR pipelines by swapping out the advantage computation. - **Strong gains on competitive programming:** On LiveCodeBench v6 with Qwen3-8B, SDPO reaches 48.8% accuracy versus 41.2% for GRPO, surpassing Claude Sonnet 4 (40.5%) and Claude Opus 4 (39.7%) on the public leaderboard. SDPO achieves GRPO's final accuracy in 4x fewer generations, with gains growing at larger model scales - suggesting self-teaching is an emergent capability. - **Effective even without rich feedback:** In standard RLVR environments with only scalar rewards, SDPO treats successful rollouts as implicit feedback for failed attempts, achieving 68.8% vs. 64.1% aggregate accuracy over GRPO on scientific reasoning and tool use benchmarks. On Chemistry with OLMo3-7B, SDPO reaches GRPO's 5-hour accuracy in just 30 minutes. - **Concise reasoning without verbosity:** SDPO produces responses that are 3-7x shorter than GRPO while achieving higher accuracy, avoiding circular reasoning patterns and filler phrases. At test time, SDPO accelerates discovery of solutions on difficult tasks by 3x compared to best-of-k sampling, enabling effective test-time self-distillation on individual questions.
Self-teacher for dense credit assignment: Instead of learning from sparse scalar rewards like GRPO, SDPO re-evaluates the modelās original attempt after conditioning on environment feedback (runtime errors, failed tests, or successful rollouts). This produces logit-level advantages at every token position, compared to GRPOās constant per-rollout advantages. The approach requires only minor changes to standard RLVR pipelines by swapping out the advantage computation.
Strong gains on competitive programming: On LiveCodeBench v6 with Qwen3-8B, SDPO reaches 48.8% accuracy versus 41.2% for GRPO, surpassing Claude Sonnet 4 (40.5%) and Claude Opus 4 (39.7%) on the public leaderboard. SDPO achieves GRPOās final accuracy in 4x fewer generations, with gains growing at larger model scales - suggesting self-teaching is an emergent capability.
Effective even without rich feedback: In standard RLVR environments with only scalar rewards, SDPO treats successful rollouts as implicit feedback for failed attempts, achieving 68.8% vs. 64.1% aggregate accuracy over GRPO on scientific reasoning and tool use benchmarks. On Chemistry with OLMo3-7B, SDPO reaches GRPOās 5-hour accuracy in just 30 minutes.
Concise reasoning without verbosity: SDPO produces responses that are 3-7x shorter than GRPO while achieving higher accuracy, avoiding circular reasoning patterns and filler phrases. At test time, SDPO accelerates discovery of solutions on difficult tasks by 3x compared to best-of-k sampling, enabling effective test-time self-distillation on individual questions.
Abstract
Large language models are increasingly post-trained with reinforcement learning in verifiable domains such as code and math. Yet, current methods for reinforcement learning with verifiable rewards (RLVR) learn only from a scalar outcome reward per attempt, creating a severe credit-assignment bottleneck. Many verifiable environments actually provide rich textual feedback, such as runtime errors or judge evaluations, that explain why an attempt failed. We formalize this setting as reinforcement learning with rich feedback and introduce Self-Distillation Policy Optimization (SDPO), which converts tokenized feedback into a dense learning signal without any external teacher or explicit reward model. SDPO treats the current model conditioned on feedback as a self-teacher and distills its feedback-informed next-token predictions back into the policy. In this way, SDPO leverages the model's ability to retrospectively identify its own mistakes in-context. Across scientific reasoning, tool use, and competitive programming on LiveCodeBench v6, SDPO improves sample efficiency and final accuracy over strong RLVR baselines. Notably, SDPO also outperforms baselines in standard RLVR environments that only return scalar feedback by using successful rollouts as implicit feedback for failed attempts. Finally, applying SDPO to individual questions at test time accelerates discovery on difficult binary-reward tasks, achieving the same discovery probability as best-of-k sampling or multi-turn conversations with 3x fewer attempts.
Get next weekās papers.
The same picks and the same summaries, in your inbox. Free, and 176 issues deep.
Subscribe on Substack