OPRD: On-Policy Reminder Distillation for Long-Horizon Coding Agents
Yuechen Luo, Ken Deng and colleagues (Tsinghua University, Shanghai AI Laboratory, NUS and independent researchers) introduce On-Policy Reminder Distillation (OPRD), which trains coding agents to act on information already in their context by distilling from a teacher that sees explicit reminders the student never sees.
Ask this paper
Diagnosis. Their analysis shows repository-level coding agents often ignore relevant information already in context, while pass/fail outcomes give sparse process supervision.
Reminders. A Global Reminder turns final-test diagnostics into trajectory-level guidance. Turn-local Reminders apply only to the next decision after the relevant environment observation.
Routing by outcome. Successful student rollouts become self-imitation targets. Failed rollouts get reverse-KL supervision from a reminder-conditioned teacher that re-scores the same student prefixes.
Results. On SWE-bench Verified, SWE-bench Pro and SWE-bench Multilingual, OPRD raises the average resolution rate by 11.3 points over the Qwen3.5-4B base model and cuts diagnosed error events from 446 to 222, a 50.2% reduction. The student runs without reminders at inference.
Abstract
Reliable repository-level coding requires agents not only to generate correct code but also to preserve constraints and respond to tool feedback across long interactions. However, our diagnostic analysis shows that agents often fail to act on relevant information already available in context, while final Pass/Fail outcomes provide only sparse process supervision. We introduce On-Policy Reminder Distillation (OPRD), which combines Multi-Scope Reminder supervision with outcome-based routing on reminder-free student rollouts. Successful rollouts provide autoregressive self-imitation targets, whereas failed rollouts receive Reminder-conditioned reverse-KL supervision from a teacher that re-evaluates the same student-generated prefixes. A Global Reminder converts final-test diagnostics into trajectory-level guidance, while Turn-local Reminders apply only to the next decision following the relevant environment observation. The teacher and student share a two-stage initialization from Coding SFT and Instruction-Following (IF) SFT. At inference, the student operates without Reminders. On SWE-bench Verified, SWE-bench Pro, and SWE-bench Multilingual, OPRD improves the benchmark-averaged task resolution rate by 11.3 points over the open-source Qwen3.5-4B base model. Under the same diagnostic protocol, it reduces the aggregate error-event count from 446 to 222, a 50.2% reduction.