What Matters in On-Policy Distillation? A Perspective on Data Efficiency and Data Selection
Zhinan Hou and colleagues at Tsinghua University and Meituan study what data on-policy distillation actually needs, and find that eight hard examples match a 17K-example baseline.
Ask this paper
1-shot on-policy distillation works: Training on a single example is consistently effective across all sampled training examples, and harder examples give larger gains.
The driver is not token entropy: The improvement tracks the longer chain-of-thought paths that hard problems naturally generate, not high-entropy tokens.
Why length helps: Longer chains keep the student aligned with the teacher over a long reasoning horizon and expose thinking patterns that short chains omit, such as reflection markers like "Alternatively".
Selection rule and result: Selecting only hard examples, including ones that completely exceed the teacher's capability, and training on eight of them matches the 17K dataset baseline across four models from 1.5B to 7B.
Abstract
On-Policy Distillation (OPD) has emerged as a widely adopted post-training paradigm for enhancing large language models in reasoning domains. However, the data-centric mechanisms in OPD remain relatively underexplored. This paper presents a empirical study of data efficiency and data selection in OPD. We begin by investigating an extreme setting: training OPD on only one example, namely 1-shot OPD. Surprisingly, we find that 1-shot OPD is consistently effective across all sampled training examples and harder examples often yield superior performance gain. We next investigate what actually drives the student model's improvement in the training data. Our analysis reveals that the improvement is not driven by high token entropy, but the longer CoT paths which hard problems naturally generate. Training on longer CoT can help maintain closer alignment with the teacher over a long reasoning horizon, and learn critical thinking patterns usually missing in short CoTs, such as reflection (e.g., ``Alternatively''). Based on these insights, we propose a simple data selection method that selects only hard examples for training, where even ``unsolvable'' examples that completely exceed the teacher's capability can be successfully used. Our experiments conducted on four models ranging from 1.5B to 7B show that training the student model on only 8 selected hard examples matches the performance of the 17K dataset baseline.