Benchmarking NN Training Algorithms (AlgoPerf)
Free while signed in. Answers cite the passages they came from.
A new benchmark for rigorously evaluating optimizers using realistic workloads.
Realistic workloads: Tests optimizers on actual production-scale tasks (ImageNet, language modeling, translation) rather than toy problems.
Wall-clock benchmarking: Evaluates optimizers on time-to-target-accuracy rather than just step counts, reflecting real training budgets.
Hyperparameter rules: Standardizes hyperparameter tuning budgets for fair cross-optimizer comparisons.
Optimizer research infrastructure: Enabled credible claims about new optimizers versus Adam and SGD - raising the bar for optimizer papers going forward.
Get next week’s papers.
The same picks and the same summaries, in your inbox. Free, and 176 issues deep.
Subscribe on Substack