Research map / Architectures and efficiency / Mixture of experts
RepLoRA: Reparameterizing Low-rank Adaptation via the Perspective of Mixture of Experts
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Analyzes LoRA as a mixture-of-experts model, shows reparameterizing LoRA matrices with lightweight MLPs reduces the sample complexity of low-rank estimation from exponential to polynomial, and proposes RepLoRA which beats LoRA on commonsense, image, video, and vision-language tasks.
Problem
Theoretical understanding of LoRA is limited, and LoRA's low-rank matrix estimation is statistically inefficient, requiring exponentially many samples to reach a given error under the MoE view.
Approach
Views each attention head's query/value LoRA updates as experts and gating in an MoE; shows that sharing structure across LoRA matrices via reparameterization (linear or nonlinear MLP over learnable diagonal matrices) turns the convergence rate from polynomially-slow-in-log to polynomial, and implements this as RepLoRA with MLPs discarded after training.
Key results
- On commonsense reasoning with LLaMA-13B, RepLoRA averages 82.9 vs LoRA 80.2 at similar parameter budget (0.99% vs 0.67%).
- On FGVC, RepLoRA averages 90.7 vs LoRA 84.8, and matches LoRA's full-data performance with only 30% of training data.
- On VTAB-1K, RepLoRA averages 75.9 vs LoRA 72.2 with ViT-B/16; on video SSv2 RepLoRA reaches 46.12 vs LoRA 38.34.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Reparameterize LoRA matrices with MLPs | LoRA · FGVC | AVG accuracy: 84.8 -> 90.7 | Helped |
| Reparameterize LoRA matrices with MLPs | LoRA · VTAB-1K | AVG accuracy: 72.2 -> 75.9 | Helped |
| Reparameterize LoRA matrices with MLPs | LoRA · Image-text understanding | AVG: 76.5 -> 78.1 | Helped |
| Reparameterize LoRA matrices with MLPs | LoRA · Video-text understanding | AVG: 83.5 -> 87.8 | Helped |
| Use non-linear rather than linear reparameterization (ablation) | RepLoRA · Multiple domains | performance | Helped |
| Use RepLoRA instead of LoRA at 1% training fraction | LoRA · FGVC | performance gap: 40.4% | Helped |
| Reduce training fraction for RepLoRA to 30% | RepLoRA · FGVC | performance relative to full-data LoRA: 30% | No clear effect |
Surprising findings
- Nonlinear reparameterization substantially outperforms the linear variant empirically.
- The theoretical rate for the non-reparameterized LoRA is claimed to require an exponential amount of data, which the paper argues is a suboptimal sample complexity.
Limitations
- Theory is simplified: it considers only the first row of one attention head and assumes query and value share low-rank matrices.
- Theoretical analysis assumes Gaussian noise and bounded support, and the rates are asymptotic.
- Training cost of the reparameterization MLPs is not characterized in the text.
Experimental setup
Datasets: commonsense reasoning suite (BoolQ, PIQA, SIQA, HellaSwag, WinoGrande, ARC-e, ARC-c, OBQA) VTAB-1K FGVC (CUB-200-2011, NABirds, Oxford Flowers, Stanford Dogs, Stanford Cars) ImageNet-21K SSv2 HMDB51 VQAv2 GQA NLVR2 MSCOCO VALUE benchmark
Benchmarks: commonsense reasoning (8 tasks) VTAB-1K FGVC SSv2 HMDB51 VQAv2 GQA NLVR2 MSCOCO captioning VALUE video-text (TVQA, How2QA, TVC, YC2C)
Metrics: top-1 accuracy PPT average accuracy CIDEr-style captioning score
Baselines: Full fine-tuning Prefix tuning LoRA DoRA Series Adapter AdaLoRA VeRA
Builds on: LoRA Mixture of Experts DoRA VeRA
Closest papers by meaning
- SDS-LoRA: Overcoming Anisotropic Gradient Scaling in Low-Rank Adaptation (NeurIPS 2026 · similarity 0.91)
- LoRA-GA: Low-Rank Adaptation with Gradient Approximation (NeurIPS 2024 · similarity 0.91)
- Flat-LoRA: Low-Rank Adaptation over a Flat Loss Landscape (ICML 2025 · similarity 0.91)
- LoRA-Pro: Are Low-Rank Adapters Properly Optimized? (ICLR 2025 · similarity 0.90)
- LoRA Training Provably Converges to a Low-Rank Global Minimum Or It Fails Loudly (But it Probably Won't Fail) (ICML 2025 · similarity 0.90)
- Beyond Higher Rank: Token-wise Input-Output Projections for Efficient Low-Rank Adaptation (NeurIPS 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.