Research map / Architectures and efficiency / Pruning and efficient tuning
AuroRA: Breaking Low-Rank Bottleneck of LoRA with Nonlinear Mapping
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AuroRA, an MLP-like LoRA variant that inserts an adaptive nonlinear layer (fixed tanh plus learnable B-spline) between low-rank projectors, matching or beating full fine-tuning with a much smaller hidden rank.
Problem
LoRA's low-rank bottleneck: closing the gap to full fine-tuning requires higher rank and more parameters, and existing linear LoRA variants remain linear in composition.
Approach
Insert an Adaptive Nonlinear Layer between LoRA's down and up projectors, combining a parameter-free tanh nonlinearity (coarse fitting) with a learnable B-spline nonlinearity (fine fitting) plus a self-projection, at a compressed hidden rank (r~=2); after training the nonlinearity is merged into a static update B·σ(A) for inference.
Key results
- On GLUE with RoBERTa-Base, AuroRA reaches 87.1 average vs 85.2 for full fine-tuning and 85.2 for LoRA, with 0.075M parameters.
- On commonsense reasoning with LLaMA3-8B, AuroRA averages 83.6 vs 80.8 for LoRA using 3.5M parameters (about 6.25% of LoRA's budget).
- On ViT-Base image classification, AuroRA averages 85.1 with 74K parameters versus 77.6 for LoRA (581K).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| introduce AuroRA nonlinear adaptation instead of LoRA | RoBERTa-Base · GLUE (6-task Avg.) | score: 85.2 -> 87.1 | Helped |
| introduce AuroRA nonlinear adaptation instead of MiLoRA | LLaMA3-8B · Commonsense reasoning (8-task Avg.) | accuracy: 81.9 -> 83.6 | Helped |
| introduce AuroRA nonlinear adaptation instead of FourierFT | ViT-Base · Image classification (8-dataset Avg.) | accuracy (%): 80.3 -> 85.1 | Helped |
| add fixed nonlinearity (ablation) | AuroRA · OxfordPets | accuracy (%): 93.3 -> 93.9 | Helped |
| add learnable nonlinearity (ablation) | AuroRA · OxfordPets | accuracy (%): 93.1 -> 93.9 | Helped |
| replace tanh with LeakyReLU (ablation) | AuroRA · StanfordCars | accuracy (%): 75.7 -> 75.6 | Hurt |
| replace tanh with Sigmoid (ablation) | AuroRA · StanfordCars | accuracy (%): 75.7 -> 75.2 | Hurt |
| use static form in both training and inference (ablation) | AuroRA · Commonsense reasoning (8-task Avg.) | accuracy: 83.6 -> 82.8 | Hurt |
| keep dynamic form for inference as well as training (ablation) | AuroRA · Commonsense reasoning (8-task Avg.) | accuracy: 83.6 -> 83.8 | No clear effect |
| keep dynamic form for inference as well as training (ablation) | AuroRA · Commonsense reasoning (8-task Avg.) | total time: $15.28$ h -> $15.60$ h | Hurt |
Surprising findings
- Removing the learnable nonlinearity hurts more than removing the fixed tanh nonlinearity in the ablation.
- Sigmoid as the fixed activation gives the lowest performance among the tested activations.
Limitations
- Experiments on larger pretrained models were not conducted due to limited computational resources.
- Theoretical guarantees rely on bounded-domain assumptions and proofs that are sketched at a high level.
Experimental setup
Datasets: GLUE Commonsense170K OxfordPets CIFAR-10 DTD EuroSAT RESISC45 StanfordCars FGVC CIFAR-100 DreamBooth
Benchmarks: GLUE BoolQ PIQA SIQA HellaSwag WinoGrande ARC-e ARC-c OBQA OxfordPets StanfordCars CIFAR10
Metrics: accuracy Matthews correlation coefficient Pearson correlation coefficient
Baselines: Full Fine-Tuning BitFit Adapter LoRA AdaLoRA DyLoRA FourierFT LoRA-drop DoRA MoSLoRA PiSSA MiLoRA
Builds on: LoRA MoSLoRA FourierFT DoRA PiSSA
Closest papers by meaning
- Uni-LoRA: One Vector is All You Need (NeurIPS 2025 · similarity 0.94)
- Beyond Higher Rank: Token-wise Input-Output Projections for Efficient Low-Rank Adaptation (NeurIPS 2025 · similarity 0.93)
- LoRA-Pro: Are Low-Rank Adapters Properly Optimized? (ICLR 2025 · similarity 0.92)
- Flat-LoRA: Low-Rank Adaptation over a Flat Loss Landscape (ICML 2025 · similarity 0.92)
- Make LoRA Great Again: Boosting LoRA with Adaptive Singular Values and Mixture-of-Experts Optimization Alignment (ICML 2025 · similarity 0.92)
- SDS-LoRA: Overcoming Anisotropic Gradient Scaling in Low-Rank Adaptation (NeurIPS 2026 · similarity 0.92)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.