Research map / Label-efficient and robust learning / Recognition and long-tail
Sparse is Enough in Fine-tuning Pre-trained Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SIFT, a gradient-based sparse fine-tuning method that updates only the top-x% gradient components (selected from the first batch) of pre-trained weights, motivated by a PAC-Bayes prior-shift analysis of loss landscapes and gradient quasi-sparsity.
Problem
Why pre-trained language models have low intrinsic dimension for fine-tuning, and how to fine-tune them efficiently without adding extra modules.
Approach
Views pre-training as a shift of the PAC-Bayes prior that yields quasi-sparse gradients; fine-tunes only the top-x% gradient-magnitude components of the pre-trained weights (sparse increment), with a hook-based memory-efficient implementation and no added modules.
Key results
- On GLUE with RoBERTa-Large at 0.8M trainable parameters, SIFT averages 88.6, matching LoRA (88.6) and close to full fine-tuning (88.9).
- On Alpaca instruction tuning, Llama-7B with SIFT reaches 40.7 MMLU and 24.1/37.2 HumanEval pass@1/pass@10 at 33B, the best HumanEval among the compared methods.
- Gradient-based SIFT selection outperforms random component selection on RTE and MRPC (avg 88.0 vs 86.5 for the best configurations).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use gradient-based component selection instead of random selection for Wq, Wk, Wv, Wo updates (ablation) | SIFT on RoBERTa-Large · RTE | accuracy: 83.8 -> 85.9 | Helped |
| Use gradient-based component selection instead of random selection for Wq, Wk, Wv, Wo updates (ablation) | SIFT on RoBERTa-Large · MRPC | Matthew’s correlation: 89.5 -> 90.4 | Helped |
| Select sparse components in Wv only rather than Wq, Wk, Wv, Wo (ablation) | SIFT on RoBERTa-Large · RTE | accuracy: 85.9 -> 85.9 | No clear effect |
| Select sparse components in Wv only rather than Wq, Wk, Wv, Wo (ablation) | SIFT on RoBERTa-Large · MRPC | Matthew’s correlation: 90.4 -> 90.4 | No clear effect |
| Select sparse components in Wq and Wv rather than Wq, Wk, Wv, Wo (ablation) | SIFT on RoBERTa-Large · RTE | accuracy: 85.9 -> 85.2 | Hurt |
| Select sparse components in Wq and Wv rather than Wq, Wk, Wv, Wo (ablation) | SIFT on RoBERTa-Large · MRPC | Matthew’s correlation: 90.4 -> 90.2 | Hurt |
| Select sparse components in Wq, Wk, Wv rather than Wq, Wk, Wv, Wo (ablation) | SIFT on RoBERTa-Large · RTE | accuracy: 85.9 -> 84.8 | Hurt |
| Select sparse components in Wq, Wk, Wv rather than Wq, Wk, Wv, Wo (ablation) | SIFT on RoBERTa-Large · MRPC | Matthew’s correlation: 90.4 -> 88.7 | Hurt |
| Select sparse components across all attention parameters rather than Wq, Wk, Wv, Wo (ablation) | SIFT on RoBERTa-Large · RTE | accuracy: 85.9 -> 86.6 | Helped |
| Select sparse components across all attention parameters rather than Wq, Wk, Wv, Wo (ablation) | SIFT on RoBERTa-Large · MRPC | Matthew’s correlation: 90.4 -> 89.0 | Hurt |
| Apply sparse updates to a smaller share of parameters by varying the sparsity budget (ablation) | SIFT on RoBERTa-Large · MRPC | performance | Mixed |
Surprising findings
- Pre-trained gradients are quasi-sparse: about 1% of components account for 99% of the gradient norm.
- Randomly selected sparse components still adapt pre-trained models reasonably well, with only a small gap to SIFT.
Limitations
- Component selection is fixed from the first batch, which may lose gradient information as training proceeds.
- Sparsity pattern is applied only to attention q/k/v/o projections chosen heuristically.
- GLUE results are on RoBERTa-Large with modest gains; SIFT does not clearly beat LoRA.
Experimental setup
Datasets: GLUE Alpaca
Benchmarks: GLUE MMLU HumanEval
Metrics: accuracy Matthews correlation Pearson correlation pass@1 pass@10
Baselines: Full fine-tuning Adapter (Adpt^H) Adapter (Adpt^P) LoRA Random selection Vanilla Llama
Builds on: LoRA Adapter Prefix-Tuning Bias-only fine-tuning PAC-Bayes bounds
Closest papers by meaning
- Linearization Explains Fine-Tuning in Large Language Models (NeurIPS 2025 · similarity 0.91)
- Dynamic Sparse No Training: Training-Free Fine-tuning for Sparse LLMs (ICLR 2024 · similarity 0.89)
- PACE: Marrying generalization in PArameter-efficient fine-tuning with Consistency rEgularization (NeurIPS 2024 · similarity 0.88)
- RoSA: Accurate Parameter-Efficient Fine-Tuning via Robust Adaptation (ICML 2024 · similarity 0.88)
- APT: Adaptive Pruning and Tuning Pretrained Language Models for Efficient Training and Inference (ICML 2024 · similarity 0.88)
- Gradient Weight-normalized Low-rank Projection for Efficient LLM Training (AAAI 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.