Stable and Low-Precision Training for Large-Scale Vision-Language Models

First page

Paper summary

Methods for accelerating and stabilizing large VLM training.

Ask this paper

Key points

Mixed-precision techniques: Introduces stable training strategies for bfloat16/float16 mixed precision of large VLMs.

Training speedup: Significantly accelerates VLM training while avoiding common instabilities (loss spikes, NaN).

Scale-friendly: Scales to the largest open-source VLMs, enabling more research at serious scale.

Infrastructure contribution: Practical infrastructure advances that benefited the entire VLM research community.

Every Monday