Research map / Architectures and efficiency / Pruning and efficient tuning
Flat-LoRA: Low-Rank Adaptation over a Flat Loss Landscape
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Flat-LoRA, which applies a Bayesian expected-loss objective with filter-wise random weight perturbations (storable as random seeds) so LoRA converges to flat minima in the full parameter space, at no extra training time.
Problem
LoRA finds minima that look flat in the low-rank space but can be sharp in the full merged parameter space, potentially hurting generalization; SAM-style fixes double training cost and memory.
Approach
Replace SAM's min-max with an expected loss under Gaussian random weight perturbations of the merged weights, scaled per filter by weight norm and by 1/input-dimension, and store only the random seed and filter norms so memory overhead is O(m) and training cost is unchanged.
Key results
- On GLUE with T5-Base, Flat-LoRA (r=16) averages 89.72% vs 89.15% for LoRA; on CLIP ViT-B/32 image classification, Flat-LoRA (r=8) averages 87.20% vs 86.64% for LoRA.
- On Llama 2-7B, Flat-LoRA (r=8) improves GSM8K from 57.47% to 60.65% and HumanEval from 24.85% to 27.93%.
- Flat-LoRA adds only 0.12GB memory and about 11 minutes training time over LoRA, versus 2x training time for LoRA+SAM.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add full-parameter Bayesian random weight perturbations to LoRA (r=8) | T5-Base · GLUE subset average | accuracy: 88.99 -> 89.47 | Helped |
| Add full-parameter Bayesian random weight perturbations to LoRA (r=8) | CLIP ViT-B/32 · Image classification average | accuracy: 86.64 -> 87.20 | Helped |
| Add full-parameter Bayesian random weight perturbations to LoRA | Llama 2-7B · GSM8K | accuracy: +3.18% | Helped |
| Add full-parameter Bayesian random weight perturbations to LoRA | Llama 2-7B · Human-Eval | PASS@1: +3.08% | Helped |
| Add full-parameter perturbations rather than perturbing only LoRA matrices A and B (ablation) | Flat-LoRA on T5-Base · MRPC | accuracy: 88.81 -> 89.59 | Helped |
| Add full-parameter perturbations rather than perturbing only LoRA matrices A and B (ablation) | Flat-LoRA on T5-Base · CoLA | accuracy: 83.19 -> 83.61 | Helped |
| Extend perturbations from linear layers to all layers (ablation) | Flat-LoRA on CLIP ViT-B/32 · CIFAR-100 | accuracy: 88.64 ± 0.23 -> 88.76 ± 0.19 | Helped |
| Set perturbation strength σ to 0.10 rather than 0 (ablation) | Flat-LoRA on CLIP ViT-B/32 · CIFAR-100 | accuracy: 87.74 ± 0.13 -> 88.65 ± 0.35 | Helped |
| Increase perturbation strength from σ=0.05 to σ=0.20 (ablation) | Flat-LoRA on Llama 2-7B · GSM8K | accuracy: 60.65 ± 0.63 -> 58.50 ± 0.85 | Hurt |
| Use full-parameter SAM instead of LoRA-parameter SAM (ablation) | LoRA on T5-Base · CoLA | accuracy: 83.31 ± 0.48 -> 83.67 ± 0.39 | Helped |
| Use full-parameter SAM instead of LoRA-parameter SAM (ablation) | LoRA on T5-Base · MRPC | accuracy: 88.98 ± 0.22 -> 89.26 ± 0.32 | Helped |
| Set perturbation strength σ to 0.05 (ablation) | Flat-LoRA on Llama 2-7B · GSM8K | accuracy: 57.47 ± 0.45 -> 60.65 ± 0.63 | Helped |
Surprising findings
- Applying SAM to LoRA parameters (A, B) gives smaller gains than applying it to full weights, and the LoRA-SAM perturbation is almost entirely within the column space of A (ratio above 0.95).
Limitations
- Perturbation strength sigma requires tuning per task (0.05 to 0.15 used), with cosine-increasing schedule.
- Gains on some tasks are modest (e.g., CIFAR-10 and MRPC differences within a few tenths of a point, often within std).
- Theoretical justification relies on a smoothing lemma and variance-amplification argument under a random-vector hypothesis rather than a full generalization bound.
Experimental setup
Datasets: GLUE (MNLI, SST2, CoLA, QNLI, MRPC) CIFAR-10 CIFAR-100 Cars SVHN DTD WizardLM MetaMathQA Code-Feedback Alpaca 3D Icons
Benchmarks: GLUE CIFAR-10 CIFAR-100 Cars SVHN DTD MT-Bench GSM8K HumanEval InstructEval CIFAR-100-C
Metrics: accuracy MT-Bench first-turn score (GPT-4 judge) pass@1 training time memory
Baselines: LoRA LoRA+SAM PiSSA LoRA-GA DoRA AdaLoRA LoRA+ Full FT oBAR nBAR LoRA-Pro GaLore
Builds on: LoRA sharpness-aware minimization random weight perturbation filter-normalized loss landscape visualization
Closest papers by meaning
- SDS-LoRA: Overcoming Anisotropic Gradient Scaling in Low-Rank Adaptation (NeurIPS 2026 · similarity 0.94)
- GraLoRA: Granular Low-Rank Adaptation for Parameter-Efficient Fine-Tuning (NeurIPS 2025 · similarity 0.93)
- The Expressive Power of Low-Rank Adaptation (ICLR 2024 · similarity 0.93)
- LoRA-GA: Low-Rank Adaptation with Gradient Approximation (NeurIPS 2024 · similarity 0.93)
- Uni-LoRA: One Vector is All You Need (NeurIPS 2025 · similarity 0.93)
- LoRA-Pro: Are Low-Rank Adapters Properly Optimized? (ICLR 2025 · similarity 0.93)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.