Stress-testing Alignment Midtraining
Sid Baines and colleagues test the assumptions behind alignment midtraining, which continues pretraining on alignment-relevant documents to encourage generalization, at up to 110B parameters and 1B midtraining tokens.
Ask this paper
Midtraining can steer motivation, in simple cases. Where post-training data is ambiguous between two motivations, midtraining resolves the ambiguity in the intended direction.
A tiny fraction of competing finetuning data erases it. Introducing a small amount of finetuning data suggesting the other motivation removes the midtraining effect entirely.
Rules need demonstrations somewhere. When a model should follow a set of rules but only a subset is demonstrated, the undemonstrated rules are not robustly learned unless demonstrations appear in midtraining or post-training.
The conclusion is stated as insufficient evidence. The authors do not claim midtraining fails, only that public evidence does not support confidence that it addresses the core difficulties of aligning powerful systems.
Abstract
When aligning frontier models through post-training techniques, it is not possible to directly demonstrate all of the behaviours we want a model to exhibit in all possible deployment environments; our model must generalise outside of the post-training distribution. One proposed solution is alignment midtraining (AMT), which continues pretraining on large volumes of alignment-relevant documents to encourage generalisation in later stages of training. Despite the prominence of AMT as an alignment approach, there is limited public evidence for its effectiveness. To resolve this, we identify several assumptions around midtraining and evaluate them across scale: up to 110 billion-parameter models and 1 billion midtraining tokens. For instance, we study a scenario where post-training data is ambiguous between two possible motivations. We find that midtraining can steer the model's motivation in simple versions of this setting. However, the presence of a tiny fraction of finetuning data which suggests a competing motivation erases the effects of AMT. We also study scenarios in which we want an AI to follow a number of rules, but only demonstrate a subset of them. We find that demonstrations must be present either in midtraining or post-training datasets for these rules to be robustly learned. Based on these and other findings, we do not believe that there is sufficient public evidence for us to confidently state that midtraining can address the core difficulties inherent in aligning powerful AI systems.