FP8-LM
Free while signed in. Answers cite the passages they came from.
Microsoft's FP8-LM demonstrates that most LLM training variables - gradients, optimizer states - can use FP8 without sacrificing accuracy.
FP8 across the pipeline: Extends FP8 training beyond forward activations to gradients and optimizer states (both moments), widening the FP8 footprint.
No hyperparameter changes: Works as a drop-in replacement for FP16/BF16 training without requiring changes to learning rates, schedules, or other hyperparameters.
Matched accuracy: Achieves accuracy indistinguishable from FP16/BF16 baselines on LLM pretraining tasks.
Efficiency gains: Delivers substantial memory and compute savings, particularly attractive for training large models on FP8-capable hardware like H100.
Get next week’s papers.
The same picks and the same summaries, in your inbox. Free, and 176 issues deep.
Subscribe on Substack