LoRA+
LoRA+ is a minimal one-line change to LoRA: use different learning rates for the down-projection (A) and up-projection (B) matrices to restore feature learning at large width.
Ask this paper
Theoretical setup: The authors show that in the infinite-width limit, identical learning rates for A and B prevent the adapter from performing proper feature learning, which LoRA inherits.
Asymmetric learning rates: Setting lr(B) = η · lr(A) with a carefully chosen ratio corrects the imbalance and recovers the correct infinite-width dynamics.
Practical gains: ~2x finetuning speedup and 1-2% absolute accuracy improvement over vanilla LoRA at the same compute cost across a broad set of tasks.
Drop-in: No changes to architecture or memory footprint - only the optimizer hyperparameter wiring changes, making LoRA+ trivial to adopt in existing PEFT pipelines.