Normalized Low-Rank Adaptation
Jiale Kang and colleagues at CUHK, Yuanshi Intelligence and Microsoft Research normalize the down-projection matrices in LoRA and report faster convergence, better stability and less forgetting at no extra cost.
Ask this paper
The observation is about initialization. LoRA sets the up-projection to zero, so early optimization dynamics are governed almost entirely by the down-projection, which makes that matrix the right place to regularize.
NoRA normalizes the down-projection during training, a change small enough to drop into an existing LoRA setup.
Initialization-only normalization also works. The same normalization applied once at initialization improves standard LoRA without repeated normalization through training, which is the cheaper of the two options.
Benefits hold across three regimes, pretraining, supervised fine-tuning and reinforcement learning, covering faster convergence, better final performance, improved training stability and reduced catastrophic forgetting.
No additional trainable parameters and no inference-time computation, which is what makes it broadly applicable rather than a specialized technique.
Abstract
While low-rank adaptation (LoRA) is widely used for parameter-efficient model adaptation, how to regularize its training dynamics for stable and effective optimization remains underexplored. Because LoRA initializes the up-projection to zero, its early optimization dynamics are largely governed by the down-projection. Building on this observation, we introduce Normalized Low-Rank Adaptation (NoRA), a simple yet effective method that normalizes the down-projection matrices during training. We further show that the same normalization can be applied only at initialization, improving standard LoRA without requiring repeated normalization throughout training. Across pretraining, supervised finetuning, and reinforcement learning, NoRA consistently accelerates convergence, improves performance and training stability, and mitigates catastrophic forgetting. These benefits require neither additional trainable parameters nor inference-time computation, making NoRA a simple and broadly applicable enhancement to LoRA.