FP8-LM
First page
Paper summary
Microsoft's FP8-LM demonstrates that most LLM training variables - gradients, optimizer states - can use FP8 without sacrificing accuracy.
Ask this paper
01
FP8 across the pipeline: Extends FP8 training beyond forward activations to gradients and optimizer states (both moments), widening the FP8 footprint.
02
No hyperparameter changes: Works as a drop-in replacement for FP16/BF16 training without requiring changes to learning rates, schedules, or other hyperparameters.
03
Matched accuracy: Achieves accuracy indistinguishable from FP16/BF16 baselines on LLM pretraining tasks.
04
Efficiency gains: Delivers substantial memory and compute savings, particularly attractive for training large models on FP8-capable hardware like H100.