GaLore
Free while signed in. Answers cite the passages they came from.
GaLore (Gradient Low-Rank Projection) reduces optimizer-state memory during LLM training while still permitting full-parameter updates, unlike LoRA-style adapters that restrict learning to a low-rank subspace.
Project gradients, not weights: Gradients are projected into a low-rank subspace for the optimizer state (momentum, variance), while weight updates themselves remain full-rank.
65.5% optimizer-state savings: Memory for optimizer state drops by up to 65.5% relative to Adam, with comparable downstream quality on pretraining LLaMA 1B and 7B architectures.
Consumer-GPU pretraining: Enables pretraining a 7B LLM on a single 24GB consumer GPU - a regime previously reserved for specialized clusters - by combining GaLore with 8-bit optimizers.
Plugs into existing stacks: GaLore is framework-agnostic and drops into standard training code without changes to model architecture or data pipelines.
Get next week’s papers.
The same picks and the same summaries, in your inbox. Free, and 176 issues deep.
Subscribe on Substack