Research map / Architectures and efficiency / Pruning and efficient tuning
The Primacy of Magnitude in Low-Rank Adaptation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LoRAM, a magnitude-driven LoRA initialization that scales deterministic DST orthogonal bases by a logarithmic spectral-gain factor and pretrained weight magnitude, matching PiSSA-style spectral initialization without SVD.
Problem
Spectral LoRA initializations (e.g., PiSSA) improve convergence and performance but add SVD computation and storage overhead, and their benefit is poorly understood theoretically.
Approach
Identifies weight update magnitude as the key driver of LoRA convergence and attributes spectral initialization gains to magnitude amplification; LoRAM mimics that magnitude with deterministic DST orthogonal bases scaled by a log-based spectral gain factor and pretrained weight statistics, avoiding decomposition.
Key results
- On LLaMA-2-7B at rank 128, LoRAM reaches 51.12 GSM8K and 31.53 MBPP versus PiSSA 51.48 and 31.07, i.e., comparable to spectral init.
- On GLUE with DeBERTa-v3-base (rank 8), LoRAM attains the best average-level results, e.g., 89.95 MRPC, 74.72 RTE, 89.93 STS-B versus PiSSA 89.21, 74.36, 88.90.
- On LLaVA, LoRAM achieves best MMMU (0.350) and ScienceQA (0.700) among LoRA variants at rank 64.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use LoRAM initialization instead of LoRA | LLaMA2-7B · GSM8K, rank 16 | accuracy (%): 31.51 ± 0.31 -> 40.32 ± 0.43 | Helped |
| Use LoRAM initialization instead of PiSSA | LLaMA2-7B · NLG tasks, rank 16 | accuracy/pass rate (%) | Helped |
| Use LoRAM initialization instead of PiSSA | LLaMA2-7B · NLG tasks, rank 128 | accuracy/pass rate (%) | Mixed |
| Use LoRAM initialization instead of PiSSA | DeBERTa-v3-base · GLUE | task score | Mixed |
| Increase Q[r] from log(r/2) to log(2r) (ablation) | LoRAM · NLG, ranks 16 and 128 | average value: 25.1 -> 25.6 | No clear effect |
| Replace DST basis with random orthogonal basis (ablation) | LoRAM · NLG tasks | average value: 25.5 -> 24.9 | No clear effect |
| Replace DST basis with Gaussian matrix (ablation) | LoRAM · NLG tasks | average value: 25.5 -> 24.8 | No clear effect |
| Use PiSSA top-r singular components instead of MiLoRA last-r components (ablation) | LoRA initialization · NLG tasks | average value: 21.1 / 21.2 -> 25.0 / 24.3 | Helped |
| Add RsLoRA scaling to LoRA-GA (ablation) | LoRA-GA · NLG tasks, rank 16 | average value: 27.4 -> 29.5 | Helped |
| Add RsLoRA scaling to LoRA-GA (ablation) | LoRA-GA · NLG tasks, rank 128 | average value: 27.4 -> 29.5 | Hurt |
| Increase LoRAM initialization magnitude Q[r] (ablation) | LoRAM · NLG tasks | task score | Helped |
| Use tracking-mode magnitudes from reference initializations (ablation) | LoRAM · NLG tasks | task score | No clear effect |
Surprising findings
- Random orthogonal or Gaussian bases perform nearly as well as DST, suggesting the basis choice has limited impact.
- Magnitude scaling has an upper bound: RsLoRA gains on LoRA-GA diminish or reverse at rank 128.
Limitations
- LoRAM mimics spectral initialization magnitudes rather than finding optimal ones.
- Different layers may benefit from tailored magnitude settings; not explored.
- Optimization dynamics and convergence properties are not explicitly analyzed.
Experimental setup
Datasets: MetaMathQA CodeFeedback Commonsense170K GLUE CC-595K LLaVA 656K mixture DreamBooth
Benchmarks: GSM8K MATH HumanEval MBPP Commonsense170K GLUE MME MMMU AI2D ChartQA OCRBench TextVQA
Metrics: accuracy pass@1 Matthews correlation Pearson correlation mean and std over three runs
Baselines: LoRA RsLoRA LoRA+ MiLoRA OLoRA PiSSA LoRA-GA CorDA
Builds on: LoRA PiSSA RsLoRA LoRA+ LoRA-GA
Closest papers by meaning
- SDS-LoRA: Overcoming Anisotropic Gradient Scaling in Low-Rank Adaptation (NeurIPS 2026 · similarity 0.91)
- LoRA Training Provably Converges to a Low-Rank Global Minimum Or It Fails Loudly (But it Probably Won't Fail) (ICML 2025 · similarity 0.91)
- LoRA-GA: Low-Rank Adaptation with Gradient Approximation (NeurIPS 2024 · similarity 0.89)
- Beyond Zero Initialization: Investigating the Impact of Non-Zero Initialization on LoRA Fine-Tuning Dynamics (ICML 2025 · similarity 0.89)
- RepLoRA: Reparameterizing Low-rank Adaptation via the Perspective of Mixture of Experts (ICML 2025 · similarity 0.89)
- Flat-LoRA: Low-Rank Adaptation over a Flat Loss Landscape (ICML 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.