ReOPD
Free while signed in. Answers cite the passages they came from.
On-policy distillation for agentic tasks is expensive because every update needs fresh student rollouts through the environment plus teacher queries at each visited history. Microsoft Research and the University of Amsterdam propose reusing pre-collected teacher trajectories instead.
Replayed prefixes: ReOPD samples a teacher trajectory, replays it as a prefix, has the student act at selected steps, and lets the teacher supply dense per-step supervision, with no new environment interaction during student training.
The prefix trap: The paper names a real pathology in multi-turn distillation. Pushing histories toward the student's own distribution makes them more relevant to the student and simultaneously drags the teacher onto states where its targets are unreliable, a two-sided shift between student occupancy and teacher reliability.
A simple control: Treating this as reliability-aware prefix distribution design, ReOPD uses a step-decaying sampling schedule that emphasizes early, lower-shift prefixes rather than trying to match the student everywhere.
Why it matters: Across math reasoning with Python and search environments, over multiple teacher and student scales, it preserves or improves accuracy, uses zero tool calls during student training, and runs at least 4 times faster per rollout, turning agent-environment interaction into a reusable offline asset.
Get next week’s papers.
The same picks and the same summaries, in your inbox. Free, and 176 issues deep.
Subscribe on Substack