Research map / Large language models / Instruction and fine-tuning
SPP: Sparsity-Preserved Parameter-Efficient Fine-Tuning for Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SPP, a parameter-efficient fine-tuning method that trains lightweight row/column multiplicative vectors on post-training-pruned LLMs so the sparsity pattern is preserved through training and merging.
Problem
Post-training pruned LLMs (SparseGPT, Wanda) lose substantial performance even at moderate sparsity, while standard PEFT methods like LoRA produce dense weights that destroy the sparsity pattern when merged.
Approach
Insert two lightweight learnable vectors (column-wise and row-wise) that multiply the frozen sparse weight element-wise, with a residual branch and zero-initialized row vector; since Hadamard products preserve zeros, merged weights keep the exact mask and sparsity ratio.
Key results
- At 75% unstructured sparsity on LLaMA-7B, Wanda+SPP reaches 41.71 average zero-shot accuracy and 21.80 perplexity vs 32.14 and 1285.24 for Wanda alone.
- With 2:4 sparsity, SPP gives around 8% average zero-shot improvement over post-training pruned models (e.g., Wanda 2:4 on LLaMA-7B from 48.34 to 55.42).
- On LLaMA-65B at 50% unstructured sparsity, Wanda+SPP reaches 67.03 average vs 66.74 for Wanda and dense 66.97.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| fine-tune Wanda-pruned weights with SPP | LLaMA 7B, unstructured 75% · 7-task zero-shot evaluation | LM-eval: 32.14 -> 41.71 | Helped |
| fine-tune Wanda-pruned weights with SPP | LLaMA 7B, 2:8 · 7-task zero-shot evaluation | LM-eval: 32.53 -> 38.61 | Helped |
| fine-tune Wanda-pruned weights with SPP | LLaMA 30B, unstructured 75% · 7-task zero-shot evaluation | LM-eval: 39.21 -> 50.33 | Helped |
| fine-tune Wanda-pruned weights with SPP | LLaMA 30B, 2:8 · 7-task zero-shot evaluation | LM-eval: 35.12 -> 43.09 | Helped |
| replace post-LoRA pruning with sparsity-preserving SPP training | LLaMA 7B, unstructured 75% · 7-task zero-shot evaluation | Average accuracy: 39.11 -> 41.71 | Helped |
| replace post-LoRA pruning with sparsity-preserving SPP training | LLaMA 7B, 2:8 · 7-task zero-shot evaluation | Average accuracy: 38.05 -> 38.61 | Helped |
| use r=16 instead of r=4 or r=8 (ablation) | Wanda+SPP on LLaMA-7B, 2:4 · 7-task zero-shot evaluation | LM-eval: 54.04; 54.87; 55.42 | Mixed |
| add Wβ (ablation) | Wanda+SPP on LLaMA-7B, 2:4 · 7-task zero-shot evaluation | LM-eval: 53.52 -> 55.42 | Helped |
| zero-initialize Wβ (ablation) | Wanda+SPP on LLaMA-7B, 2:4 · 7-task zero-shot evaluation | LM-eval: 53.52 -> 55.42 | Helped |
| use r=16 instead of r=4 or r=8 (ablation) | Wanda+SPP on LLaMA-7B, unstructured 50% · 7-task zero-shot evaluation | LM-eval: 57.86; 56.39; 58.13 | Mixed |
| add Wβ (ablation) | Wanda+SPP on LLaMA-7B, unstructured 50% · 7-task zero-shot evaluation | LM-eval: 57.81 -> 58.13 | Helped |
| zero-initialize Wβ (ablation) | Wanda+SPP on LLaMA-7B, unstructured 50% · 7-task zero-shot evaluation | LM-eval: 57.59 -> 58.13 | Helped |
Surprising findings
- SPP-trained sparse models in two cases exceed their dense counterparts' zero-shot average.
- Zero-initializing both added vectors blocks gradient flow, leaving them at zero; initializing only the row vector to zero works best.
Limitations
- Fine-tuning schemes result in out-of-memory errors for models above 30B parameters on 8 A100-80GB GPUs in some variants.
- Still a notable gap to dense models on MMLU for sparse models.
- Only fixed-mask fine-tuning; mask is not updated during training.
Experimental setup
Datasets: Stanford-Alpaca
Benchmarks: BoolQ RTE HellaSwag WinoGrande ARC-e ARC-c OBQA MMLU WikiText perplexity
Metrics: zero-shot accuracy average accuracy 5-shot accuracy perplexity
Baselines: SparseGPT Wanda LoRA* DS⊘T
Builds on: LoRA Wanda SparseGPT Megatron-LM
Closest papers by meaning
- PAT: Pruning-Aware Tuning for Large Language Models (AAAI 2025 · similarity 0.92)
- APT: Adaptive Pruning and Tuning Pretrained Language Models for Efficient Training and Inference (ICML 2024 · similarity 0.92)
- SparseLLM: Towards Global Pruning of Pre-trained Language Models (NeurIPS 2024 · similarity 0.92)
- Dynamic Sparse No Training: Training-Free Fine-tuning for Sparse LLMs (ICLR 2024 · similarity 0.91)
- Pruning Large Language Models with Semi-Structural Adaptive Sparse Training (AAAI 2025 · similarity 0.91)
- The Journey Matters: Average Parameter Count over Pre-training Unifies Sparse and Dense Scaling Laws (ICLR 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.