Fine-Tuning Language Models with Just Forward Passes (MeZO)

No backpropagation: Uses a memory-efficient zeroth-order SGD algorithm that requires only forward passes, eliminating the memory overhead of backprop.

Inference-like memory: Fine-tunes large LLMs with the same memory footprint as inference - democratizes full-parameter fine-tuning.

Comparable quality: Reaches comparable quality to backpropagation-based fine-tuning on many tasks despite using only forward passes.

Memory-constrained tuning: Opens new possibilities for fine-tuning huge models on modest hardware by trading compute for memory.