LoRA+
Free while signed in. Answers cite the passages they came from.
LoRA+ is a minimal one-line change to LoRA: use different learning rates for the down-projection (A) and up-projection (B) matrices to restore feature learning at large width.
Theoretical setup: The authors show that in the infinite-width limit, identical learning rates for A and B prevent the adapter from performing proper feature learning, which LoRA inherits.
Asymmetric learning rates: Setting lr(B) = η · lr(A) with a carefully chosen ratio corrects the imbalance and recovers the correct infinite-width dynamics.
Practical gains: ~2x finetuning speedup and 1-2% absolute accuracy improvement over vanilla LoRA at the same compute cost across a broad set of tasks.
Drop-in: No changes to architecture or memory footprint - only the optimizer hyperparameter wiring changes, making LoRA+ trivial to adopt in existing PEFT pipelines.
Get next week’s papers.
The same picks and the same summaries, in your inbox. Free, and 176 issues deep.
Subscribe on Substack