Optimizer Memory Schedules for Outscaling the Overtraining Axis
Katie Everett (MIT CSAIL) and Shikai Qiu (NYU) show that optimizer rankings and optimal hyperparameters change substantially as the training horizon extends, and argue that overtraining factor belongs on the axis list for optimizer evaluation.
Ask this paper
Sweep: Four optimizers (AdamW, Muon, SOAP, ADANA) across models from 51M to 253M parameters and overtraining factors from 1x to 256x, with the base learning rate swept at every setting.
Hyperparameters move with horizon: The preferred learning rate schedule can reverse across the overtraining axis, the best weight decay coefficient scales roughly as sqrt(OT), and longer horizons favor longer fixed memory.
ADANA's advantage is not a tuning artifact: It persists after AdamW's fixed memory is tuned separately at each horizon. Log-time weight decay and momentum cooldown add gains that compound as training lengthens, and with that treatment ADANA outscales AdamW with an exponent advantage close to the DANA theory prediction on power-law random features.
Matrix-preconditioned methods behave differently: Muon and SOAP give roughly constant token-efficiency advantages over AdamW across most of the range. ADANA starts behind both, closes the gap, passes Muon, and becomes competitive with SOAP at the highest overtraining factors.
Abstract
We investigate how optimizers scale across the overtraining axis and show that relative optimizer performance and optimal hyperparameters change substantially with training horizon. In particular, we study how matrix-preconditioned methods (Muon and SOAP) and a momentum-scheduled method (ADANA) scale relative to AdamW. We compare these four optimizers across models from 51M to 253M parameters and overtraining (OT) factors from 1x to 256x, sweeping the base learning rate at every setting. The preferred learning rate schedule can reverse across the overtraining axis, the best weight decay coefficient scales approximately as sqrt(OT), and longer horizons generally favor longer fixed memory. ADANA's scaling advantage over AdamW persists after tuning AdamW's fixed memory separately at each horizon. Log-time weight decay and momentum cooldown provide substantial gains for ADANA that compound as training increases. With this treatment, ADANA outscales AdamW with an exponent advantage close to that predicted by DANA theory on power-law random features. Muon and SOAP instead provide roughly constant token-efficiency advantages over AdamW across most of the measured range, although SOAP may gain further at the highest overtraining factors. ADANA begins behind both matrix-preconditioned optimizers but closes the gaps as training increases, surpassing Muon and becoming competitive with SOAP at our highest OT factors. These results establish training horizon as an essential axis for optimizer evaluation and design.