Research map / Architectures and efficiency / Pruning and efficient tuning
SVFT: Parameter-Efficient Fine-Tuning with Singular Vectors
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SVFT, a PEFT method that updates a frozen weight matrix as a sparse, trainable combination of its own singular vectors (U M V^T), training only the sparse coefficients.
Problem
Parameter-efficient fine-tuning methods such as LoRA and its variants trade performance for parameter count and their updates ignore the geometry of the weight matrix they modify, leaving a gap to full fine-tuning.
Approach
Parameterize the weight update as U M V^T, where U and V are the frozen singular vectors of the pre-trained matrix and M is a trainable sparse matrix with a fixed sparsity pattern (Plain, Banded, Random, Top-k), so the perturbation structure depends on W itself.
Key results
- On GSM-8K with Gemma-2B, SVFT-R (d=16, 6.35M params) reaches 50.03% vs Full-FT 52.69% (about 96% recovery) and beats DoRA r=16 (44.27%).
- On commonsense reasoning with Gemma-7B, SVFT-B d=8 (9.8M params) averages 83.35% vs LoRA r=32 83.69% (68.8M params).
- SVFT-P with about 0.2M parameters outperforms BOFT and LoRA r=1 (about 6x more parameters) by up to 18% relative on GSM-8K with Gemma-2B.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SVFT Random d=16 instead of DoRA rank 16 | Gemma-2B fine-tuning · GSM-8K | accuracy: 35.25 -> 50.03 | Helped |
| Use SVFT Random d=16 instead of LoRA rank 32 | Gemma-7B fine-tuning · GSM-8K | accuracy: 76.57 -> 76.81 | Helped |
| Use SVFT Banded d=8 instead of LoRA rank 32 | Gemma-7B fine-tuning · Commonsense reasoning average | accuracy: 83.69 -> 83.35 | Hurt |
| Use SVFT Plain instead of LoRA rank 1 | DeBERTaV3-base fine-tuning · GLUE average | average score: 88.23 -> 88.54 | Helped |
| Use SVFT Banded d=8 instead of Full-FT | ViT-B fine-tuning · CIFAR-100 | accuracy: 85.35 -> 85.69 | Helped |
| Use SVFT Banded d=2 instead of LoRA rank 8 | ViT-L fine-tuning · Resisc45 | accuracy: 79.62 -> 79.70 | Helped |
| Change the SVFT structure from Banded to Random at 3.3M parameters (ablation) | Gemma-2B SVFT · GSM-8K | accuracy: 46.47 -> 47.76 | Helped |
| Change the SVFT structure from Banded to Random at 3.3M parameters (ablation) | Gemma-2B SVFT · MATH | accuracy: 16.04 -> 15.98 | No clear effect |
| Change the SVFT structure from Banded to Top-k at 3.3M parameters (ablation) | Gemma-2B SVFT · GSM-8K | accuracy: 46.47 -> 48.00 | Helped |
| Change the SVFT structure from Banded to Top-k at 3.3M parameters (ablation) | Gemma-2B SVFT · MATH | accuracy: 16.04 -> 15.80 | Hurt |
| Truncate singular vectors and values to rank 128 with d=64 instead of full rank 2048 (ablation) | Gemma-2B SVFT · GSM-8K | accuracy: 40.34 -> 0.98 | Hurt |
| Truncate singular vectors and values to rank 1536 with d=2 instead of full rank 2048 (ablation) | Gemma-2B SVFT · GSM-8K | accuracy: 40.34 -> 25.01 | Hurt |
Surprising findings
- Adapting only the singular-vector-based coefficients for the plain variant (diagonal M) is competitive with LoRA r=1 despite far fewer parameters.
- Full fine-tuning overfits on larger models (Gemma-7B, LLaMA-3-8B), giving sub-optimal results relative to PEFT methods.
Limitations
- Requires computing SVD and storing both left and right singular vectors, roughly doubling memory relative to LoRA.
- Sparsity patterns are heuristic (Plain, Banded, Random, Top-k) rather than learned or principled, and their effectiveness varies by task.
Experimental setup
Datasets: MetaMathQA-40K commonsense-15K GLUE CIFAR-100 Food101 RESISC45 Flowers102 ImageNet-21k
Benchmarks: GSM-8K MATH BoolQ PIQA SIQA HellaSwag Winogrande ARC-easy ARC-challenge OpenBookQA GLUE CIFAR-100
Metrics: accuracy matthews correlation pearson correlation
Baselines: Full-FT LoRA DoRA BOFT VeRA
Builds on: LoRA DoRA VeRA BOFT PiSSA
Closest papers by meaning
- Parameter Efficient Fine-tuning via Explained Variance Adaptation (NeurIPS 2025 · similarity 0.88)
- Sensitivity-Aware Efficient Fine-Tuning via Compact Dynamic-Rank Adaptation (CVPR 2025 · similarity 0.88)
- Parameter-Efficient Fine-Tuning with Discrete Fourier Transform (ICML 2024 · similarity 0.88)
- Expanding Sparse Tuning for Low Memory Usage (NeurIPS 2024 · similarity 0.88)
- Lessons and Insights from a Unifying Study of Parameter-Efficient Fine-Tuning (PEFT) in Visual Recognition (CVPR 2025 · similarity 0.88)
- Structured Unrestricted-Rank Matrices for Parameter Efficient Finetuning (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.