Quiet Feature Learning
Free while signed in. Answers cite the passages they came from.
Researchers reveal a hidden learning phenomenon in Transformers trained on algorithmic tasks. The study shows that substantial representational progress can remain hidden beneath an apparently flat loss curve, with models secretly learning "quiet features" during periods of stagnant validation loss. - **Quiet features discovery:** During extended periods where validation loss appears stagnant, models learn intermediate computational representations that encode algorithmic steps but don't immediately reduce task loss. - **Phase transitions:** Training on ten foundational algorithmic tasks reveals pronounced phase transitions that deviate from typical power-law scaling, challenging conventional understanding of model training dynamics. - **Causal necessity:** Through ablation studies, the team demonstrated that individual quiet features are causally necessary for eventual task performance, not merely correlated artifacts. - **Training implications:** The findings challenge reliance on cross-entropy loss as the sole training indicator, suggesting that richer diagnostics are needed to properly monitor model learning progress.
Quiet features discovery: During extended periods where validation loss appears stagnant, models learn intermediate computational representations that encode algorithmic steps but don’t immediately reduce task loss.
Phase transitions: Training on ten foundational algorithmic tasks reveals pronounced phase transitions that deviate from typical power-law scaling, challenging the conventional understanding of model training dynamics.
Causal necessity: Through ablation studies, the team demonstrated that individual quiet features are causally necessary for eventual task performance, not merely correlated artifacts.
Training implications: The findings challenge reliance on cross-entropy loss as the sole training indicator, suggesting that richer diagnostics are needed to properly monitor model learning progress.
Abstract
We train Transformer-based language models on ten foundational algorithmic tasks and observe pronounced phase transitions in their loss curves that deviate from established power-law scaling trends. Over large ranges of compute, the validation loss barely improves, then abruptly decreases. Probing the models' internal representations reveals that quiet features are learned prior to any decrease in task loss. These quiet features represent intermediate algorithmic computations that do not by themselves improve the output loss. Ablation experiments demonstrate that individual quiet features are causally necessary for task performance. Our results demonstrate that substantial representational progress can remain hidden beneath an apparently flat loss curve, challenging the prevailing use of cross-entropy as a proxy for learning and motivating richer diagnostics for monitoring model training.
Get next week’s papers.
The same picks and the same summaries, in your inbox. Free, and 176 issues deep.
Subscribe on Substack