On-Policy Self-Training: Data-Free Self-Improvement in Large Language Models
Leon Sick, Lukas Hoyer, Yannick Strümpler, Christian Reisswig, Timo Ropinski and Federico Tombari (Google, Ulm University) introduce On-Policy Self-Training (OPST), which post-trains a model on problems and reasoning it generates itself from coarse topic descriptions only.
Ask this paper
Method. The model writes its own questions, answers and privileged reasoning from topic descriptors. A teacher copy conditioned on the generated answer supervises the student on-policy, instead of training directly on the answer.
Gemma 4 E2B. Without thinking, MMLU rises from 37.5 to 47.7 (+10.2) and HellaSwag from 48.7 to 56.2; SFT and GRPO on the same self-generated data gain at most 2.1 points.
Filtering. Pseudo-verification and a consensus gate add 5.0 MMLU points over unfiltered OPST, and conditioning on the full rationale beats answer-only conditioning (47.7 vs 44.6).
Scope and limits. Gains extend to multi-domain curricula and vision-language QA with unlabeled images. The method only amplifies existing knowledge and consistent errors can still enter training.
Abstract
On-policy self-distillation enables post-training foundation models without relying on stronger external teachers. However, existing approaches rely on ground-truth question-answer pairs, human-curated problem sets, or other supervised signals. In this work, we reformulate on-policy self-distillation as a self-training framework, driven by self-generated question-answer pairs and privileged reasoning demonstrations. We push supervision to the limit: guided solely by coarse, high-level topic descriptions, the model synthesizes its own curriculum of queries and privileged reasoning traces without access to any human-curated prompts or answers. We find that self-distillation on fully synthetic privileged information enables a model to improve across multiple domains while retaining its capabilities on domains it is not self-trained on. We demonstrate the generality of this framework across text and multimodal domains, achieving consistent self-improvement on standard language benchmarks and Visual Question Answering. Our findings demonstrate that foundation models possess rich latent capabilities that can be amplified without external task supervision or curated problem distributions.