Research map / Architectures and efficiency / Pruning and efficient tuning

RoSA: Rotational Sparse Adaptation for Memory-Efficient Fine-Tuning

NeurIPS 2026 · Architectures and efficiency › Pruning and efficient tuning · approach: Adapter-based parameter-efficient tuning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes RoSA, a fine-tuning schedule that freezes an input-side prefix of LLM layers, keeps the output layer active, and rotates a trainable block of layers over depth (combined with NanoAdam sparse optimization) to cut peak memory while staying near full fine-tuning accuracy.

Problem

Full fine-tuning of LLMs needs too much optimizer-state and activation memory for commodity GPUs, while PEFT methods restrict the trainable subspace to a fixed low-dimensional set throughout training.

Approach

Freeze a consecutive input-side prefix, keep an output-side anchor always trainable, and rotate one consecutive block of later layers through training epochs, with sparse (NanoAdam) optimizer states inside the active block; this bounds instantaneous memory and shortens backprop while cumulatively updating many more parameters.

Key results

• On Llama-2-7B, RoSA + NanoAdam reduces peak VRAM from 78.5 GiB (full AdamW) to 34.9 GiB while reaching 79.61% average commonsense accuracy vs 79.99% for FFT.
• At matched density (rho=0.1), rotation cuts peak memory from 46.3 GiB (full NanoAdam) to 33.9 GiB.
• At matched memory (~30 GiB), RoSA with group size 6 reaches 78.50% vs 70.05% for LISA on Llama-2-7B.

Surprising findings

• Core-only rotation over Layers 8-23 performs almost identically to full RoSA (79.56% vs 79.61%).
• Training loss steps down at rotation boundaries without spikes despite cold-starting optimizer states on revisited groups.

Limitations

• Current implementation still runs the full forward pass; activation caching benefits are not realized and caching would add memory and data-loading costs.
• RoSA can require more epochs and longer wall-clock time than dense fine-tuning when the latter fits in memory.
• Evaluated only on models up to 7B and a small set of tasks; no instruction tuning or multimodal evaluation.

Experimental setup

Datasets: BoolQ PIQA SIQA HellaSwag WinoGrande ARC-Easy ARC-Challenge OpenBookQA GSM8K

Benchmarks: BoolQ PIQA SIQA HellaSwag WinoGrande ARC-Easy ARC-Challenge OpenBookQA GSM8K

Metrics: accuracy average accuracy peak VRAM wall-clock training time

Baselines: AdamW full fine-tuning full-model NanoAdam LoRA DoRA LISA GaLore BAdam static layer subsets

Builds on: NanoAdam BAdam LISA LoRA DoRA

Closest papers by meaning

• RoSA: Accurate Parameter-Efficient Fine-Tuning via Robust Adaptation (ICML 2024 · similarity 0.89)
• Parameter Efficient Fine-tuning via Explained Variance Adaptation (NeurIPS 2025 · similarity 0.88)
• Sheared Backpropagation for Fine-tuning Foundation Models (CVPR 2024 · similarity 0.87)
• SparseLoRA: Accelerating LLM Fine-Tuning with Contextual Sparsity (ICML 2025 · similarity 0.87)
• APT: Adaptive Pruning and Tuning Pretrained Language Models for Efficient Training and Inference (ICML 2024 · similarity 0.87)
• S$^{2}$FT: Efficient, Scalable and Generalizable LLM Fine-tuning by Structured Sparsity (NeurIPS 2024 · similarity 0.87)