Spacing Effect for Generalization

Researchers from Tsinghua University investigate how the spacing effect - a well-documented learning principle where spaced intervals between training improve retention - can enhance generalization in both biological and artificial neural networks. - **Bio-inspired hypothesis:** The spacing effect promotes integration of input and innate variations during learning, enabling better generalization to novel but related scenarios. The researchers test this by introducing bio-inspired spacing mechanisms into artificial neural networks. - **Spaced dropout implementation:** The approach implements structured dropout where probability varies periodically to introduce structured neuronal variability during training. Test accuracy follows a U-shaped trend, indicating optimal performance at intermediate variation strengths. - **Cross-architecture validation:** Performance gains from spaced dropout are demonstrated across different network architectures and benchmark datasets, showing the approach generalizes beyond specific model types. - **Flatter loss landscapes:** Theoretical and empirical analyses show that spacing effect benefits stem from convergence to flatter loss landscapes during stochastic gradient descent, resulting in better real-world performance and stronger resistance to noisy data.