Research map / Large language models / Instruction and fine-tuning

Dynamic Sparse No Training: Training-Free Fine-tuning for Sparse LLMs

ICLR 2024 · Large language models › Instruction and fine-tuning · approach: Efficient LLM fine-tuning systems · 5 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes DSnoT, a training-free method that refines the masks of one-shot pruned LLMs via iterative pruning-and-growing guided by reconstruction error, without any weight updates or backpropagation.

Problem

One-shot pruning of LLMs loses substantial performance at high sparsity, and fine-tuning sparse LLMs to recover it is too expensive in compute and data.

Approach

Adapts dynamic sparse training's prune-and-grow cycle as a training-free, gradient-free mask refinement that minimizes layer-wise dense-vs-sparse output reconstruction error, using expectation and variance of the per-weight reconstruction change for growing and a transformed Wanda metric for pruning.

Key results

• At 60% sparsity on LLaMA-V1-7B, DSnoT improves Wanda perplexity from 10.69 to 10.22 and SparseGPT from 10.41 to 9.65 on WikiText-2.
• At 70% sparsity on LLaMA-7B, DSnoT outperforms Wanda by 26.79 perplexity (abstract claim), reducing perplexity from 88.84 to 62.05.
• DSnoT refines Wanda-pruned LLaMA-V1-7B in 4.3 seconds versus 4 hours for LoRA fine-tuning, with perplexity 7.12 vs 6.87 at 50% sparsity.

Measured effects reported in the paper

Surprising findings

• Pruning the weights that would most reduce reconstruction error hurt performance significantly, indicating outlier-channel weights must be preserved.

Limitations

• Gains shrink or vanish in some zero-shot tasks, with no consistent superiority across pruning methods
• Evaluation focuses on perplexity and six zero-shot tasks, with no instruction-following or generation benchmarks
• Performance at very high sparsity (e.g., 80-90%) remains very high perplexity

Experimental setup

Datasets: C4 WikiText-2

Benchmarks: WikiText-2 perplexity PIQA StoryCloze ARC-Easy ARC-Challenge HellaSwag OpenBookQA

Metrics: perplexity zero-shot accuracy

Baselines: Magnitude pruning SparseGPT Wanda LoRA

Builds on: Wanda SparseGPT Dynamic Sparse Training Optimal Brain Surgeon

Closest papers by meaning

• R-Sparse: Rank-Aware Activation Sparsity for Efficient LLM Inference (ICLR 2025 · similarity 0.92)
• Dynamic Low-Rank Sparse Adaptation for Large Language Models (ICLR 2025 · similarity 0.92)
• The Journey Matters: Average Parameter Count over Pre-training Unifies Sparse and Dense Scaling Laws (ICLR 2025 · similarity 0.92)
• SPP: Sparsity-Preserved Parameter-Efficient Fine-Tuning for Large Language Models (ICML 2024 · similarity 0.91)
• Pruning Large Language Models with Semi-Structural Adaptive Sparse Training (AAAI 2025 · similarity 0.91)
• SparseLLM: Towards Global Pruning of Pre-trained Language Models (NeurIPS 2024 · similarity 0.91)