Research map / Architectures and efficiency / Pruning and efficient tuning
Expanding Sparse Tuning for Low Memory Usage
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SNELL, a sparse-tuning PEFT method that sparsifies a kernelized low-rank (LoRA-style) adaptation matrix via competition-based thresholding, cutting memory while matching or beating prior sparse and LoRA methods on VTAB-1k and FGVC.
Problem
Sparse tuning for PEFT achieves strong accuracy but has high memory usage because the whole weight matrix is stored as optimizer parameters and tunable weight indexes must be stored, making it no cheaper than full fine-tuning on large models.
Approach
Decompose the sparsified adaptation matrix into two learnable low-rank matrices, merge them with nonlinear kernel functions (instead of an inner product) to raise the merged matrix's rank, and sparsify it with a competition-based dynamic threshold on absolute values so no binary mask needs storing.
Key results
- On FGVC with supervised ViT-B/16, SNELL-32 reaches 91.8% mean accuracy vs 90.7% for the prior best (SSF), and on VTAB-1k 76.9% (SNELL-32) vs 76.4% for SPT-LoRA/SPT-Adapter in the main table.
- Memory on ViT-H/14 is about 50% of full fine-tuning, and SNELL's memory overhead over LoRA-8 is 0.038 to 0.076 of SNELL-8 memory across ViT-B/L/H.
- On LLaMA-2-7B commonsense reasoning, SNELL-32 averages 78.7 vs 77.6 for LoRA-32.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace the LoRA inner product with a piecewise linear kernel (ablation) | LoRA on VTAB-1k · VTAB-1k mean accuracy, ranks 8, 16, and 32 | mean accuracy: 75.1 -> 75.6; 74.9 -> 75.4; 74.2 -> 75.2 | Helped |
| Use Sigmoid or RBF kernels instead of the piecewise linear kernel (ablation) | Kernelized LoRA fine-tuning · VTAB-1k | performance | Hurt |
| Add sparse tuning to kernelized LoRA (ablation) | KLoRA on VTAB-1k · VTAB-1k mean accuracy, ranks 8, 16, and 32 | mean accuracy: +1.0; +1.4; +1.9 | Helped |
| Increase learnable-matrix rank (ablation) | KLoRA on VTAB-1k · VTAB-1k mean accuracy, ranks 8 to 32 | mean accuracy: 73.2 -> 72.7 | Hurt |
| Increase learnable-matrix rank (ablation) | SNELL on VTAB-1k · VTAB-1k mean accuracy, ranks 8 to 32 | mean accuracy: 74.2 -> 74.6 | Helped |
| Use a LoRA merged adaptation matrix rather than a full-rank matrix for sparsification (ablation) | Sparse tuning on VTAB-1k · VTAB-1k mean accuracy, s=0.9 | mean accuracy: 74.4 -> 65.7 | Hurt |
| Replace LoRA with kernelized LoRA before sparsification (ablation) | Sparse tuning on VTAB-1k · VTAB-1k mean accuracy, s=0.9 | mean accuracy: 65.7 -> 73.9 | Helped |
| Use predefined fixed weight masks instead of competition-based dynamic masks (ablation) | KLoRA-8 on FGVC · FGVC mean accuracy | mean accuracy: 90.3 -> 89.4 | Hurt |
| Replace the inner product with a piecewise linear kernel (ablation) | LoRA on VTAB-1k · Natural, Specialized, and Structured groups | accuracy: Natural: 80.8 -> 80.8; Specialized: 84.9 -> 85.5; Structured: 59.6 -> 60.5 | Mixed |
| Use SNELL sparse tuning with kernelized low-rank adaptation | ViT-B/16 on FGVC · FGVC mean accuracy | mean accuracy: 90.7 -> 91.8 | Helped |
| Use SNELL instead of SPT-LoRA | ViT-B/16 on VTAB-1k · VTAB-1k mean accuracy | mean accuracy: 76.4 -> 76.9 | Helped |
| Use SNELL instead of LoRA-8 | ViT-L/16 on VTAB-1k · VTAB-1k mean accuracy | mean accuracy: 73.7 -> 75.3 | Helped |
Surprising findings
- Sparsifying a plain LoRA merged matrix performs much worse than a full-rank matrix (65.7 vs 74.4 mean VTAB accuracy), while kernelized LoRA recovers near full-rank performance.
Limitations
- Training time is higher than LoRA due to recomputing the merged adaptation matrix during backpropagation.
- Sparsity ratio must be searched per task, and the parameter volume is not reported.
Experimental setup
Datasets: FGVC (CUB-200-2011, NABirds, Oxford Flowers, Stanford Dogs, Stanford Cars) VTAB-1k (19 tasks) Commonsense reasoning benchmark (BoolQ, PIQA, SIQA, HellaSwag, WinoGrande, ARC, OBQA)
Benchmarks: FGVC VTAB-1k commonsense reasoning (LLaMA-2-7B)
Metrics: top-1 accuracy mean accuracy peak memory usage (MB) training time (s/img)
Baselines: Full fine-tuning LoRA-8 LoRA-16 SPT-LoRA SPT-Adapter VPT-Deep VPT-Shallow Adapter-8 Adapter-32 SSF Bias Partial-1
Builds on: LoRA SPT DyN Sparse tuning (binary mask) VPT
Closest papers by meaning
- Sensitivity-Aware Efficient Fine-Tuning via Compact Dynamic-Rank Adaptation (CVPR 2025 · similarity 0.89)
- SVFT: Parameter-Efficient Fine-Tuning with Singular Vectors (NeurIPS 2024 · similarity 0.88)
- PELA: Learning Parameter-Efficient Models with Low-Rank Approximation (CVPR 2024 · similarity 0.87)
- Unleashing the Power of Meta-tuning for Few-shot Generalization Through Sparse Interpolated Experts (ICML 2024 · similarity 0.87)
- SDS-LoRA: Overcoming Anisotropic Gradient Scaling in Low-Rank Adaptation (NeurIPS 2026 · similarity 0.87)
- S$^{2}$FT: Efficient, Scalable and Generalizable LLM Fine-tuning by Structured Sparsity (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.