GaLore
GaLore (Gradient Low-Rank Projection) reduces optimizer-state memory during LLM training while still permitting full-parameter updates, unlike LoRA-style adapters that restrict learning to a low-rank subspace.
Ask this paper
Project gradients, not weights: Gradients are projected into a low-rank subspace for the optimizer state (momentum, variance), while weight updates themselves remain full-rank.
65.5% optimizer-state savings: Memory for optimizer state drops by up to 65.5% relative to Adam, with comparable downstream quality on pretraining LLaMA 1B and 7B architectures.
Consumer-GPU pretraining: Enables pretraining a 7B LLM on a single 24GB consumer GPU - a regime previously reserved for specialized clusters - by combining GaLore with 8-bit optimizers.
Plugs into existing stacks: GaLore is framework-agnostic and drops into standard training code without changes to model architecture or data pipelines.