Research map / Architectures and efficiency / Pruning and efficient tuning
Sparse Structure Search for Delta Tuning
Shengding Hu, Zhen Zhang, Ning Ding, Yadao Wang, Yasheng Wang, Zhiyuan Liu, Maosong Sun
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Search sparse structure with S3Delta-M (1.39% params) instead of manual BitFit (8.13% params) | T5-large delta tuning · GLUE | average score: 86.90 -> 87.17 | Improved |
| Search with S3Delta-M instead of manual LoRA (r=1) at fewer parameters | T5-large delta tuning · GLUE | average score: 86.84 -> 87.17 | Improved |
| Search with S3Delta-M instead of manual BitFit | T5-large delta tuning · SuperGLUE excluding COPA | average score: 82.41 -> 83.10 | Improved |
| Search with S3Delta-M instead of manual LoRA (r=1) | T5-large delta tuning · SuperGLUE excluding COPA | average score: 81.88 -> 83.10 | Improved |
| Narrow the search space from Mix to LoRA only (S3Delta-L vs S3Delta-M) (ablation) | S3Delta · GLUE | average score: 87.17 -> 86.91 | Hurt |
| Narrow the search space from Mix to LoRA only (S3Delta-L vs S3Delta-M) (ablation) | S3Delta · SuperGLUE | average score: 81.30 -> 79.26 | Hurt |
| Search LoRA positions (S3Delta-L) instead of uniform LoRA (r=1) | LoRA · SuperGLUE excluding COPA | average score: 81.88 -> 82.46 | Improved |
| Reduce trainable parameters of S3Delta-M from 1.39% to 0.35% (ablation) | S3Delta-M · GLUE | average score: 87.17 -> 86.43 | Hurt |
| Reduce trainable parameters of S3Delta-M from 1.39% to 0.35% (ablation) | S3Delta-M · SuperGLUE excluding COPA | average score: 83.10 -> 81.24 | Hurt |
| Use S3Delta-M structure instead of random structures at extremely low budgets (0.086%) | S3Delta-M · RTE, MultiRC, MRPC | FT performance recovered: 96.8%,98.7%,97.5% | Improved |
| Increase trainable budget to 5.6% | all methods · MultiRC, MRPC | score | No clear effect |
| Replace L0 regularization with shifted global sigmoid for sparsity control (ablation) | S3Delta · MRPC, RTE, MultiRC | score | Improved |
Closest papers by meaning
- S$^{2}$FT: Efficient, Scalable and Generalizable LLM Fine-tuning by Structured Sparsity (NeurIPS 2024 · similarity 0.87)
- SparseLoRA: Accelerating LLM Fine-Tuning with Contextual Sparsity (ICML 2025 · similarity 0.85)
- DARE the Extreme: Revisiting Delta-Parameter Pruning For Fine-Tuned Models (ICLR 2025 · similarity 0.84)
- DeltaPhi: Physical States Residual Learning for Neural Operators in Data-Limited PDE Solving (NeurIPS 2025 · similarity 0.83)
- Maintaining Structural Integrity in Parameter Spaces for Parameter Efficient Fine-tuning (ICLR 2025 · similarity 0.83)
- Trust the Direction, Search the Step: Zero-and-First-Order Methods for LLM Fine-Tuning (NeurIPS 2026 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
