Research map / Architectures and efficiency / Pruning and efficient tuning
PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PiSSA, a PEFT method that initializes low-rank adapters from the principal singular values/vectors of the pretrained weight and freezes the residual, plus QPiSSA with 4-bit quantized residual, outperforming LoRA/QLoRA on NLG and NLU tasks.
Problem
LoRA initializes adapters with Gaussian noise and zeros, so gradients are small or uninformative early on, causing slow convergence and suboptimal fits; QLoRA also leaves quantization error unreduced.
Approach
Apply SVD to the pretrained weight W, initialize trainable adapters A,B from the top-r principal singular components, and freeze the residual W^res built from the remaining components; quantizing only W^res reduces quantization error relative to QLoRA.
Key results
- Gemma-7B fine-tuned with PiSSA reaches 77.78% on GSM8K vs. 74.53% for LoRA (kaiming), and PiSSA beats LoRA on all NLG tasks across LLaMA 2-7B, Mistral-7B, Gemma-7B.
- On GLUE with DeBERTa-v3-base, PiSSA reaches 89.83 average vs. 88.62 for LoRA-K and 89.46 for AdaLoRA.
- QPiSSA reduces quantization error by about 19-20% on average relative to QLoRA (which reduces 0%), and QPiSSA on LLaMA-3-70B reaches 86.05% GSM8K vs. 81.73% for QLoRA (from abstract).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| initialize trainable adapters from principal singular values and vectors instead of Gaussian/zero initialization | LoRA fine-tuning · GSM8K, MATH, HumanEval, MBPP, and MT-Bench | benchmark score | Helped |
| use principal instead of medium or minor singular values and vectors to initialize adapters (ablation) | PiSSA · GSM8K and MATH | accuracy | Helped |
| increase PiSSA rank from 64 to 128 (ablation) | PiSSA · Quantization error reduction ratio (AVG), LLaMA 3-70B | quantization error reduction ratio: 13.4 -> 20.3 | Helped |
| increase QPiSSA SVD iterations from 1 to 5 (ablation) | QPiSSA rank 64 · Quantization error reduction ratio (AVG), LLaMA 3-8B | quantization error reduction ratio: 7.7 -> 14.3 | Helped |
| increase Fast SVD niter from 1 to 16 at rank 8 (ablation) | PiSSA Fast SVD initialization · Training set | training loss: 0.3044 -> 0.2946 | Helped |
| replace exact SVD with Fast SVD (ablation) | PiSSA initialization · Training set | training loss: 0.2945; 0.2946 | No clear effect |
| train in FP32 instead of BF16 (ablation) | full fine-tuning · GSM8K, LLaMA-2-7B | accuracy: 63.15 -> 68.31 | Helped |
| train in FP32 instead of BF16 (ablation) | full fine-tuning · GSM8K, Mistral-7B and LLaMA-3-8B | accuracy: 73.09 -> 65.88; 81.96 -> 75.44 | Hurt |
| combine PiSSA with AdaLoRA improvements (ablation) | PiSSA · GSM8K, LLaMA-3-8B | accuracy: 76.75±0.036 -> 78.59±0.199 | Helped |
| combine PiSSA with DoRA (ablation) | PiSSA · GSM8K, LLaMA-3-8B | accuracy: 76.75±0.036 -> 77.51±0.257 | Helped |
| quantize the residual matrix to 4-bit instead of quantizing the full base model | QLoRA · GSM8K, LLaMA-3-70B | accuracy: 81.73% -> 86.05% | Helped |
| increase QPiSSA SVD iterations from 1 to 5 (ablation) | QPiSSA rank 64 · Quantization error reduction ratio (AVG), LLaMA-3-70B | quantization error reduction ratio: 8.6 -> 13.4 | Helped |
Surprising findings
- Fine-tuning with BF16 vs FP32 changes results substantially and the direction is inconsistent across models, so the authors could not identify which precision is better.
- AdaLoRA does not actually perform SVD on any matrix despite its SVD-like terminology.
Limitations
- No theoretical explanation for PiSSA's advantage over LoRA is provided.
- Only language models are studied; applicability to convolutional or vision models is untested.
- Results of BF16 vs FP32 and rank sensitivity are mixed, and PiSSA benefits from a multi-iteration SVD for stronger quantization error reduction at extra cost.
Experimental setup
Datasets: MetaMathQA CodeFeedback WizardLM-Evol-Instruct GLUE GSM8K MATH HumanEval MBPP MT-Bench
Benchmarks: GSM8K MATH HumanEval MBPP MT-Bench GLUE
Metrics: accuracy average score quantization error reduction ratio training loss
Baselines: LoRA QLoRA LoftQ Full FT AdaLoRA DoRA BitFit HAdapter PAdapter
Builds on: LoRA QLoRA LoftQ AdaLoRA DoRA
Closest papers by meaning
- Uni-LoRA: One Vector is All You Need (NeurIPS 2025 · similarity 0.90)
- QA-LoRA: Quantization-Aware Low-Rank Adaptation of Large Language Models (ICLR 2024 · similarity 0.90)
- LoRA-One: One-Step Full Gradient Could Suffice for Fine-Tuning Large Language Models, Provably and Efficiently (ICML 2025 · similarity 0.90)
- Bayesian Low-rank Adaptation for Large Language Models (ICLR 2024 · similarity 0.90)
- AltLoRA: Towards Better Gradient Approximation in Low-Rank Adaptation with Alternating Projections (NeurIPS 2025 · similarity 0.89)
- VB-LoRA: Extreme Parameter Efficient Fine-Tuning with Vector Banks (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.