Research map / Architectures and efficiency / Pruning and efficient tuning
RoSA: Accurate Parameter-Efficient Fine-Tuning via Robust Adaptation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RoSA, a PEFT method that trains a low-rank adapter plus a highly sparse adapter (mask chosen after LoRA warm-up) on frozen LLaMA2-7B weights, with sparse GPU kernels and a quantized variant QRoSA.
Problem
LoRA-type PEFT methods often fail to match full fine-tuning accuracy on complex generative tasks (math reasoning, SQL generation), while full fine-tuning of LLMs is memory-prohibitive.
Approach
Inspired by robust PCA, jointly train a low-rank adapter and a highly sparse adapter on top of frozen pretrained weights, with the sparse support chosen from gradients after a short LoRA warm-up, plus custom sparse GPU kernels and compatibility with quantized base weights.
Key results
- On GSM8k with a 80M budget, RoSA (r=16, d=0.6%) reaches 32.2% after one epoch vs 28.4% LoRA r=16 (40M) and FFT 32.3%; extended training gives RoSA 38.6% vs FFT 38.8%.
- On ViGGO extended training, RoSA (r=16, d=0.6%) reaches 97.3% at 163M budget, above FFT's 95.0%.
- RoSA fits in about 20-21 GB GPU memory on a single 24GB RTX 3090 at 40-160M budgets, roughly 40-100x fewer trainable parameters than FFT; QRoSA reaches 33.1% on GSM8k with under 12 GB.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| combine low-rank and sparse adapters rather than use LoRA alone | LLaMA2-7B fine-tuning · GSM8K 1 Epoch | accuracy: 28.4 -> 32.8 | Helped |
| combine low-rank and sparse adapters rather than use LoRA alone | LLaMA2-7B fine-tuning · ViGGO 1 Epoch | accuracy: 90.5 -> 95.2 | Helped |
| combine low-rank and sparse adapters rather than use LoRA alone | LLaMA2-7B fine-tuning · SQL 1 Epoch | accuracy: 89.1 vs 88.9; RoSA r=4,d=0.45%: 89.7 | Mixed |
| combine low-rank and sparse adapters | LLaMA2-7B fine-tuning with 4-bit double-quantized base weights · GSM8K | accuracy: 29.8 -> 33.1 | Helped |
| combine low-rank and sparse adapters | LLaMA2-7B fine-tuning with 4-bit double-quantized base weights · SQL | accuracy: 89.4 -> 88.6 | Hurt |
| use warm-up LoRA gradient-magnitude mask instead of initialization gradient-magnitude mask (ablation) | RoSA · GSM8K 1 Epoch | accuracy: 30.10 -> 32.16 | Helped |
| use warm-up LoRA gradient-magnitude mask instead of initialization gradient-RPCA mask (ablation) | RoSA · GSM8K 1 Epoch | accuracy: 29.87 -> 32.16 | Helped |
| use warm-up LoRA gradient-magnitude mask instead of weight-RPCA mask (ablation) | RoSA · GSM8K 1 Epoch | accuracy: 30.71 -> 32.16 | Helped |
| use warm-up LoRA gradient-magnitude mask instead of random mask (ablation) | RoSA · GSM8K 1 Epoch | accuracy: 30.25 -> 32.16 | Helped |
| replace LoRA with RoSA for OpenPlatypus instruction tuning | LLaMA2-7B · MMLU 5-shot (OpenPlatypus) | accuracy: 49.92 -> 46.54 | Hurt |
| replace LoRA with RoSA for Alpaca instruction tuning | LLaMA2-7B · MMLU 5-shot (Alpaca) | accuracy: 45.8 -> 46.52 | Helped |
| use RoSA rather than LoRA | LLaMA2-7B fine-tuning, 80M parameter budget · runtime | batch/second: 0.1149 -> 0.0595 | Hurt |
Surprising findings
- In single-epoch training, RoSA surpasses full fine-tuning on GSM8k, ViGGO and SQL.
- RoSA does not outperform LoRA on MMLU after instruction tuning on OpenPlatypus or Alpaca, where tasks are simpler.
Limitations
- RoSA is about 1.7-2x slower than LoRA in the current implementation due to sparse kernel overheads and FP32 sparse operators.
- Advantage over LoRA is small or absent on simpler instruction-tuning tasks (MMLU results).
- Evaluation is limited to LLaMA2-7B and three specialized datasets; hyperparameter tuning is required to find the best low-rank/sparse split.
Experimental setup
Datasets: GSM8k ViGGO SQL generation (Spider-based formation from prior work) OpenPlatypus Alpaca
Benchmarks: GSM8k ViGGO SQL generation MMLU
Metrics: accuracy peak memory (GB) batches per second
Baselines: full fine-tuning (FFT) LoRA sparse adaptation (SpA) DSEE FISH Mask QLoRA IA3
Builds on: LoRA FISH Mask DSEE Sputnik QLoRA
Closest papers by meaning
- Linearization Explains Fine-Tuning in Large Language Models (NeurIPS 2025 · similarity 0.89)
- RoSA: Rotational Sparse Adaptation for Memory-Efficient Fine-Tuning (NeurIPS 2026 · similarity 0.89)
- Sparse is Enough in Fine-tuning Pre-trained Large Language Models (ICML 2024 · similarity 0.88)
- Data Efficient Adaptation in Large Language Models via Continuous Low-Rank Fine-Tuning (NeurIPS 2025 · similarity 0.88)
- Parameter-Efficient Fine-Tuning with Discrete Fourier Transform (ICML 2024 · similarity 0.88)
- GraLoRA: Granular Low-Rank Adaptation for Parameter-Efficient Fine-Tuning (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.