The Boundary of Neural Network Trainability is Fractal

Fractal boundaries: Zooming into the hyperparameter plane (e.g. learning rate × init scale), the trainable region has a self-similar boundary that stays fractal over more than ten decades of zoom.

Universal across setups: Observed for every tested neural network configuration including deep linear networks, suggesting the fractal structure is a property of the training dynamics itself.

Edge-of-stability: The best-performing hyperparameter choices tend to sit right at the edge of stability, consistent with recent "edge-of-stability" observations in deep learning theory.

Implication: Hyperparameter search is more analogous to exploring a chaotic dynamical system than a smooth optimization surface, which has implications for both practitioners and learning theorists.