Self-Improving Pretraining
Free while signed in. Answers cite the passages they came from.
Self-Improving Pretraining is a new pretraining paradigm from Meta FAIR that replaces standard next-token prediction with sequence-level generation guided by an existing post-trained model acting as both a suffix rewriter and a suffix judge. The approach addresses quality, safety, and factuality issues at pretraining time rather than deferring them to post-training, yielding large gains across all three dimensions. - **Suffix rewriting and judging framework:** The method segments pretraining data into prefix-suffix chunks. A post-trained teacher model rewrites low-quality or unsafe suffixes into superior training targets, while a separate judge scores candidate completions (original suffixes, rewrites, and policy rollouts) to provide rewards for online RL training via online DPO or reward-filtered NLL. - **Strong continual pretraining gains:** When applied to continual pretraining of Llama2 1.4B, the method achieves an 86.3% generation quality win rate over the baseline, a 36.2% relative improvement in factuality (42.3 to 57.6 average score), and an 18.5% relative improvement in safety (76.9 to 91.1 average score), while also improving standard evaluation benchmarks. - **From-scratch pretraining improvements:** Training from scratch on RedPajama yields a 31.1% absolute gain in generation quality win rate and safety evaluations improving from 85.2 to 97.5, demonstrating that embedding quality signals early in pretraining is highly effective. - **Scaling with rollouts:** Performance improves consistently with more rollouts during online DPO training (tested from 1 to 16), and the model naturally transitions from relying on suffix rewrites early in training to preferring its own high-quality rollouts as training progresses.
Suffix rewriting and judging framework: The method segments pretraining data into prefix-suffix chunks. A post-trained teacher model rewrites low-quality or unsafe suffixes into superior training targets, while a separate judge scores candidate completions (original suffixes, rewrites, and policy rollouts) to provide rewards for online RL training via online DPO or reward-filtered NLL.
Strong continual pretraining gains: When applied to continual pretraining of Llama2 1.4B, the method achieves an 86.3% generation quality win rate over the baseline, a 36.2% relative improvement in factuality (42.3 to 57.6 average score), and an 18.5% relative improvement in safety (76.9 to 91.1 average score), while also improving standard evaluation benchmarks.
From-scratch pretraining improvements: Training from scratch on RedPajama yields a 31.1% absolute gain in generation quality win rate, and safety evaluations, improving from 85.2 to 97.5, demonstrating that embedding quality signals early in pretraining is highly effective.
Scaling with rollouts: Performance improves consistently with more rollouts during online DPO training (tested from 1 to 16), and the model naturally transitions from relying on suffix rewrites early in training to preferring its own high-quality rollouts as training progresses.
Abstract
Ensuring safety, factuality and overall quality in the generations of large language models is a critical challenge, especially as these models are increasingly deployed in real-world applications. The prevailing approach to addressing these issues involves collecting expensive, carefully curated datasets and applying multiple stages of fine-tuning and alignment. However, even this complex pipeline cannot guarantee the correction of patterns learned during pretraining. Therefore, addressing these issues during pretraining is crucial, as it shapes a model's core behaviors and prevents unsafe or hallucinated outputs from becoming deeply embedded. To tackle this issue, we introduce a new pretraining method that streams documents and uses reinforcement learning (RL) to improve the next K generated tokens at each step. A strong, post-trained model judges candidate generations -- including model rollouts, the original suffix, and a rewritten suffix -- for quality, safety, and factuality. Early in training, the process relies on the original and rewritten suffixes; as the model improves, RL rewards high-quality rollouts. This approach builds higher quality, safer, and more factual models from the ground up. In experiments, our method gives 36.2% and 18.5% relative improvements over standard pretraining in terms of factuality and safety, and up to 86.3% win rate improvements in overall generation quality.
Get next week’s papers.
The same picks and the same summaries, in your inbox. Free, and 176 issues deep.
Subscribe on Substack