Shaping & Monitoring Models through their Internal Representations
Thomas Fel, Matthew Kowal, Matt Feiszli and colleagues at Goodfire, with senior authors Thomas McGrath and Owen Lewis, propose Mechanistic Shaping, which trains models against their own internal representations so they follow rules about which concepts they express and which concepts they rely on.
Ask this paper
Two kinds of rules. Expression rules constrain what the model outputs or represents, for example no harmful content. Reliance rules constrain what it uses to decide, for example not a candidate's gender when screening a résumé.
Breadth. Tested on vision, medical imaging and language, at up to 32B parameters, across classification, SFT, DPO and GRPO. Targeted behaviors drop substantially while task performance is preserved.
Monitorability. Training against a representation can change it, so the authors define a hierarchy of monitor classes that separates a concept that escapes one probe from a concept that has actually been removed.
Three circumvention mechanisms. The concept drifts away from the monitor, is erased from the layer, or persists as a nonlinear encoding in higher-order statistics. Each can be detected by a matching monitor.
Practical result. Refitted linear probes recover shaped concepts in most settings, and the original probe stays reliable whenever gradients do not flow through it.
Abstract
We propose Mechanistic Shaping, a unified framework for shaping models through their internal representations. Mechanistic Shaping is applied during training so that models follow specifications (rules) about which concepts they express, measured via their outputs or internal representations (e.g., no harmful content), and also which concepts they rely on, i.e., what they use to make their decisions (e.g., not a candidate's gender when screening a résumé). Across multiple domains (vision, medical imaging, language), scales (up to 32B parameters), and training paradigms (classification, SFT, DPO, GRPO), we find that Mechanistic Shaping substantially reduces targeted behaviors while preserving task performance. Since training against a representation can also modify it, we ask whether shaped concepts remain monitorable, a question that depends on the class of monitors considered. We therefore introduce a hierarchy of such classes, which separates a concept that escapes one probe from a concept that is truly gone. We characterize three geometric mechanisms of circumvention: the concept can drift away from the monitor, be erased from the layer, or persist in higher-order statistics as a nonlinear encoding, which we explain through a product-code construction. Although other mechanisms may arise in different settings or at larger scales, these three account for all of our observations, and each can be detected by a matching monitor. In practice, we find that shaped concepts largely remain monitorable: refitted linear probes recover the concept in most settings, and the original probe remains reliable whenever gradients do not flow through it. Taken together, these results point to the potential of using interpretability as a tool for training, and call for a correspondingly rigorous science of interpretability-based training.