Research map / Architectures and efficiency / Pruning and efficient tuning
Spectral Adapter: Fine-Tuning in Spectral Space
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Spectral Adapter, which fine-tunes the top-r singular vectors of pretrained weights either additively (A) or via Cayley-parameterized orthogonal rotation (R), improving rank capacity over LoRA, multi-adapter fusion, and parameter efficiency.
Problem
Parameter-efficient fine-tuning methods such as LoRA ignore the spectral information of pretrained weight matrices; the paper asks whether tuning the spectral (SVD) space improves PEFT in accuracy, parameter efficiency, and multi-adapter fusion.
Approach
Take the SVD of each pretrained weight and fine-tune only the top-r left/right singular vectors, either additively (Spectral Adapter^A, which doubles rank capacity versus LoRA at equal parameters) or by an exact orthogonal rotation parameterized by Cayley transform (Spectral Adapter^R, with only two r×r trainable matrices); orthogonal spectral slices also separate concepts for multi-adapter fusion
Key results
- On GLUE with DeBERTaV3-base, Spectral Adapter^A (r=24, 0.72% params) reaches 88.03 average vs 86.47 for LoRA and 87.10 for AdaLoRA.
- On GSM8K with Mistral 7B (r=8, 0.16% params), Spectral Adapter^A scores 49.73 vs 44.81 for LoRA and 43.82 for DoRA.
- Spectral Adapter^R reaches recognizable custom vase concepts with about 20k trainable parameters and can go down to 1k, while LoRA-style baselines start at tens of thousands or more.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Spectral Adapter^A instead of AdaLoRA | DeBERTaV3-base · GLUE average | accuracy: 87.10 -> 88.03 | Helped |
| Use Spectral Adapter^A instead of LoRA | Mistral 7B · GSM8K | accuracy: $44.81\pm 1.37$ -> $49.73\pm 1.38$ | Helped |
| Tune the top eighth singular-vector ranks instead of the bottom eighth (ablation) | Spectral Adapter^A · DeBERTaV3-base CoLA | validation score | Helped |
| Tune the bottom eighth singular-vector ranks instead of the top eighth (ablation) | Spectral Adapter^A · DeBERTaV3-base CoLA | validation score | Hurt |
| Use bottom ranks instead of top ranks (ablation) | Spectral Adapter^A · Diffusion single-object generation | generation quality | Hurt |
| Use top singular-vector columns for concept-specific adapters | Spectral Adapter^A multi-adapter fusion · Chilloutmix multi-object generation | image alignment | Helped |
| Use Spectral Adapter^R for tuning with small parameter budgets | Chilloutmix diffusion model · Custom vase and chair generation | generation quality | Helped |
Surprising findings
- Tuning the top singular directions outperforms tuning the bottom ones, contrary to the LoRA paper's finding that the LoRA update avoids top singular directions.
Limitations
- Choice of tuning the top spectral space is justified theoretically only under simple settings and is not fully understood.
- Fine-tuning only attention layers or other components is left unexplored.
- SVD time grows with model size.
Experimental setup
Datasets: GLUE GSM8K Orca Math
Benchmarks: GLUE GSM8K
Metrics: accuracy average GLUE score CLIP image alignment score CLIP text alignment score runtime GPU storage
Baselines: LoRA DoRA OFT AdaLoRA SVDiff LiDB VeRA Gradient Fusion Orthogonal Adaptation FedAvg
Builds on: LoRA OFT DoRA SVDiff
Closest papers by meaning
- F-Adapter: Frequency-Adaptive Parameter-Efficient Fine-Tuning in Scientific Machine Learning (NeurIPS 2025 · similarity 0.89)
- SVFT: Parameter-Efficient Fine-Tuning with Singular Vectors (NeurIPS 2024 · similarity 0.87)
- Efficient Adaptation of Pre-trained Vision Transformer via Householder Transformation (NeurIPS 2024 · similarity 0.87)
- Parameter-Efficient Fine-Tuning with Discrete Fourier Transform (ICML 2024 · similarity 0.86)
- Asymmetry in Low-Rank Adapters of Foundation Models (ICML 2024 · similarity 0.86)
- Linearization Explains Fine-Tuning in Large Language Models (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.