Comparing Sparsified MLPs and Transcoders: Many Transcoder Features Are Neurons in Disguise
Stepan Shabalin and Mateusz Piotrowski (Anthropic Fellows, Georgia Tech) show that a large share of active transcoder features in late layers are near-copies of individual MLP neurons, and that only those features correspond stably to the underlying MLP.
Ask this paper
Neuron features. A neuron feature is a transcoder feature whose weights align strongly with one MLP neuron. These account for a substantial fraction of active late-layer features across 13 open-source transcoders.
Stability test. Non-neuron features vary more between training runs, do not match the MLP's Jacobian, and move less from their random initialization.
Implication. MLP neurons are a stronger basis for circuit analysis than the polysemanticity view assumes, which questions how much transcoders add.
Open problem. The authors call for methods that attribute the non-neuron structure the model actually uses.
Abstract
MLP neurons in transformer language models are widely understood to be polysemantic, motivating transcoders - sparse, learned approximations of MLP layers - as interpretable replacement models for mechanistic analysis. However, recent work shows that MLP neurons can be used to recover sparse, interpretable circuits (Arora et al. 2026, Ye et al. 2025), raising the question of whether transcoders are necessary for circuit analysis. We define neuron features as transcoder features whose weights are strongly aligned with individual MLP neurons, and show that they account for a substantial fraction of active features in late layers across 13 open-source transcoders. Only neuron features have a stable correspondence with something in the underlying MLP: other features vary more between training runs, do not match the MLP's Jacobian, and change less significantly from their random initialization. Our findings reframe MLP neurons as a stronger basis for interpretability than commonly assumed, and call for better methods of attributing the non-neuron structure that is used by the model.