Explaining Grokking
DeepMind advances our understanding of grokking, predicting and confirming two novel phenomena that test their theory.
Ask this paper
Ungrokking: A model can go from perfect generalization back to memorization when trained further on a smaller dataset below a critical threshold - the first demonstration of this reverse effect.
Semi-grokking: A randomly initialized network trained on the critical dataset size shows a grokking-like transition but partial, rather than the sharp full-grokking curve.
Theoretical predictions: These behaviors were predicted from theory before being demonstrated empirically - a rare example of predictive rather than post-hoc explanation in deep learning.
Generalization theory: Advances understanding of when and why neural networks transition from memorization to generalization, bridging empirical observation with principled prediction.