Self-Improving Pretraining
Free while signed in. Answers cite the passages they came from.
Self-Improving Pretraining is a new pretraining paradigm from Meta FAIR that replaces standard next-token prediction with sequence-level generation guided by an existing post-trained model acting as both a suffix rewriter and a suffix judge. The approach addresses quality, safety, and factuality issues at pretraining time rather than deferring them to post-training, yielding large gains across all three dimensions. - **Suffix rewriting and judging framework:** The method segments pretraining data into prefix-suffix chunks. A post-trained teacher model rewrites low-quality or unsafe suffixes into superior training targets, while a separate judge scores candidate completions (original suffixes, rewrites, and policy rollouts) to provide rewards for online RL training via online DPO or reward-filtered NLL. - **Strong continual pretraining gains:** When applied to continual pretraining of Llama2 1.4B, the method achieves an 86.3% generation quality win rate over the baseline, a 36.2% relative improvement in factuality (42.3 to 57.6 average score), and an 18.5% relative improvement in safety (76.9 to 91.1 average score), while also improving standard evaluation benchmarks. - **From-scratch pretraining improvements:** Training from scratch on RedPajama yields a 31.1% absolute gain in generation quality win rate and safety evaluations improving from 85.2 to 97.5, demonstrating that embedding quality signals early in pretraining is highly effective. - **Scaling with rollouts:** Performance improves consistently with more rollouts during online DPO training (tested from 1 to 16), and the model naturally transitions from relying on suffix rewrites early in training to preferring its own high-quality rollouts as training progresses.
Post-trained model as rewriter and judge: The strong model rewrites suffixes during pretraining, then judges rollouts of the in-training model against both the rewrite and the original. Safety, factuality, and quality become reward signals rather than post-hoc filters, which lets the policy internalize the targets early.
Sequence generation from the start: The policy is trained to generate sequences directly under reward, not to predict the next token. This shifts the inductive bias toward producing the kinds of continuations the judge rewards, which matters most on long-form generation where token-level losses lose discriminative signal.
Concrete gains across the board: 36.2% relative gain in factuality, 18.5% in safety, and up to 86.3% win rate in generation quality over standard pretraining. The safety and factuality numbers are large enough to suggest these properties are easier to install during pretraining than to retrofit after the fact.
Why it matters: The post-trained models you already have can be used to pretrain the next ones better. That is a recursive improvement loop at the pretraining layer, which is where the largest behavioral commitments actually get locked in.
Abstract
Ensuring safety, factuality and overall quality in the generations of large language models is a critical challenge, especially as these models are increasingly deployed in real-world applications. The prevailing approach to addressing these issues involves collecting expensive, carefully curated datasets and applying multiple stages of fine-tuning and alignment. However, even this complex pipeline cannot guarantee the correction of patterns learned during pretraining. Therefore, addressing these issues during pretraining is crucial, as it shapes a model's core behaviors and prevents unsafe or hallucinated outputs from becoming deeply embedded. To tackle this issue, we introduce a new pretraining method that streams documents and uses reinforcement learning (RL) to improve the next K generated tokens at each step. A strong, post-trained model judges candidate generations -- including model rollouts, the original suffix, and a rewritten suffix -- for quality, safety, and factuality. Early in training, the process relies on the original and rewritten suffixes; as the model improves, RL rewards high-quality rollouts. This approach builds higher quality, safer, and more factual models from the ground up. In experiments, our method gives 36.2% and 18.5% relative improvements over standard pretraining in terms of factuality and safety, and up to 86.3% win rate improvements in overall generation quality.
Get next week’s papers.
The same picks and the same summaries, in your inbox. Free, and 176 issues deep.
Subscribe on Substack