Shaping Capabilities with Token-Level Data Filtering
Free while signed in. Answers cite the passages they came from.
Researchers from Anthropic and Stanford show that filtering pretraining data at the token level is a highly effective, scalable, and robust approach for selectively removing undesired capabilities from language models. Using medical knowledge removal as a proxy task, token-level filtering Pareto dominates document-level filtering and achieves a 7,000x compute slowdown on the target domain for 1.8B parameter models - while preserving capabilities in related fields. - **Token filtering beats document filtering:** Inspired by data attribution research showing individual tokens vary in their influence on model capabilities, the authors filter tokens rather than whole documents. This achieves the same reduction in undesired capabilities with lower cost to benign ones, since document filtering removes many useful tokens alongside harmful ones. Sweeping across classifier thresholds on 521M models confirms token filtering Pareto dominates document filtering. - **Effectiveness scales with compute:** Training models from 61M to 1.8B parameters, the authors find filtering gets more effective at larger scales. For 1.8B models, token removal causes a 7,000x effective compute slowdown on the forget domain versus just 30x for document filtering. On multiple choice medical benchmarks, filtered models score near chance, while retaining full performance on biology, STEM, and non-STEM evaluations. - **10x more robust than unlearning:** Token-filtered models are 10x more robust to adversarial finetuning attacks than state-of-the-art unlearning methods. This addresses a key limitation of post-hoc approaches - once a capability exists in a base model, it is extremely hard to remove, but preventing it from forming during pretraining is far more durable. - **Compatibility with alignment and SAE-based labeling:** Surprisingly, models trained with token filtering generalize to refusal training better than unfiltered baselines, countering concerns that filtered models cannot be properly aligned on removed domains. The authors also introduce a novel pipeline using sparse autoencoders to label tokens and distill cheap, high-quality classifiers, showing that filtering remains effective even with noisy labels given sufficient compute.
Token filtering beats document filtering: Inspired by data attribution research showing individual tokens vary in their influence on model capabilities, the authors filter tokens rather than whole documents. This achieves the same reduction in undesired capabilities with lower cost to benign ones, since document filtering removes many useful tokens alongside harmful ones. Sweeping across classifier thresholds on 521M models confirms that token filtering Pareto dominates document filtering.
Effectiveness scales with compute: Training models from 61M to 1.8B parameters, the authors find filtering gets more effective at larger scales. For 1.8B models, token removal causes a 7,000x effective compute slowdown on the forget domain versus just 30x for document filtering. On multiple choice medical benchmarks, filtered models score near chance, while retaining full performance on biology, STEM, and non-STEM evaluations.
10x more robust than unlearning: Token-filtered models are 10x more robust to adversarial finetuning attacks than state-of-the-art unlearning methods. This addresses a key limitation of post-hoc approaches - once a capability exists in a base model, it is extremely hard to remove, but preventing it from forming during pretraining is far more durable.
Compatibility with alignment and SAE-based labeling: Surprisingly, models trained with token filtering generalize to refusal training better than unfiltered baselines, countering concerns that filtered models cannot be properly aligned on removed domains. The authors also introduce a novel pipeline using sparse autoencoders to label tokens and distill cheap, high-quality classifiers, showing that filtering remains effective even with noisy labels given sufficient compute.
Abstract
Current approaches to reducing undesired capabilities in language models are largely post hoc, and can thus be easily bypassed by adversaries. A natural alternative is to shape capabilities during pretraining itself. On the proxy task of removing medical capabilities, we show that the simple intervention of filtering pretraining data is highly effective, robust, and inexpensive at scale. Inspired by work on data attribution, we show that filtering tokens is more effective than filtering documents, achieving the same hit to undesired capabilities at a lower cost to benign ones. Training models spanning two orders of magnitude, we then demonstrate that filtering gets more effective with scale: for our largest models, token filtering leads to a 7000x compute slowdown on the forget domain. We also show that models trained with token filtering can still be aligned on the forget domain. Along the way, we introduce a methodology for labeling tokens with sparse autoencoders and distilling cheap, high-quality classifiers. We also demonstrate that filtering can be robust to noisy labels with sufficient pretraining compute.
Get next week’s papers.
The same picks and the same summaries, in your inbox. Free, and 176 issues deep.
Subscribe on Substack