Data-free On-policy Distillation
Gengsheng Li and colleagues at the Institute of Automation, Chinese Academy of Sciences and Tencent find that on-policy distillation barely depends on its training data, and propose Data-free On-policy Distillation (DF-OPD), in which the teacher writes its own training questions.
Ask this paper
Eight prompts are enough: On the two most common teacher-student pairings, eight prompts match a 17k-problem dataset, and three datasets with several-fold differences in difficulty and teacher-student KL give nearly identical training curves.
Why: The unit of data in OPD is the state a prompt leads to. One prompt keeps exposing new teacher corrections as sampling continues, so extra prompts add little after about eight.
Reasoning mode transfers: Replacing math prompts with competitive programming prompts still recovers over 90% of the in-domain gain, which indicates OPD transfers the teacher's way of reasoning rather than knowledge tied to the data.
Data-free distillation: In DF-OPD the teacher writes the questions under a simple prompt with no filtering. It matches or beats real data, and in multi-teacher distillation 1k self-generated questions close 98.5% of the teacher-student gap against 96.6% with about 7k real post-training examples.
Abstract
On-policy distillation (OPD) has become a standard component of frontier post-training pipelines, yet how much its training data actually contributes has gone largely unexamined. On the two teacher-student pairings most common in practice, we find OPD almost indifferent to its data: eight prompts already match a 17k-problem dataset, and three independently built datasets whose difficulty and teacher-student KL differ several-fold produce nearly indistinguishable training curves. Two causes account for this. First, the unit of data in OPD is the state a prompt leads to, not the prompt itself: a single prompt keeps exposing new teacher correction as sampling continues, while the marginal value of additional prompts collapses after eight. Second, replacing mathematics with competitive programming still recovers over ninety percent of the in-domain gain, indicating that OPD transfers the teacher's mode of reasoning rather than knowledge related to the data. We take this to its limit with Data-free On-policy Distillation (DF-OPD), in which the teacher writes its own training questions under a simple prompt -- no external data, no filtering -- leaving a system of just two policies. DF-OPD matches and even surpasses real data, and the questions it produces track the teacher's own post-training data on three key diagnostics of training dynamics, which other real datasets do not. Applied to multi-teacher distillation, where the (prompt, domain) pairs normally have to be derived from post-training data that is often out of reach, 1k self-generated questions close 98.5% of the available headroom, even surpassing the 96.6% reached with 7k real examples. Moreover, together these results invite a reassessment of the role data plays in OPD.