Research map / Large language models / Language, safety and interpretability
An Efficient Pruner for Large Language Model with Theoretical Guarantee
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Reformulates one-shot layer-wise LLM pruning as an ℓ0-penalized problem solved by a monotone accelerated iterative hard thresholding (mAIHT) method, with convergence and risk-bound theory and LLaMA-7B experiments.
Problem
One-shot pruning of LLMs typically relies on heuristic approaches lacking theoretical guarantees, and prior optimization-based methods (ADMM, FISTA) either lack guarantees or relax the ℓ0 problem to ℓ1.
Approach
Casts layer-wise pruning as ℓ0-penalized least squares solved by proximal gradient with hard thresholding, adds a monotone Nesterov-style acceleration with a loss-monitoring safeguard, and adaptively tunes the penalty λ to hit a target sparsity.
Key results
- On WikiText2 at 0.5 sparsity, mAIHT reaches perplexity 7.0720 versus 7.0826 for ADMM, 7.2397 for SparseGPT and 7.2588 for Wanda.
- At 0.1 sparsity mAIHT (5.6928) is slightly worse than ADMM (5.6925) but better than other baselines.
- Across sparsity levels 0.2 to 0.5 on seven zero-shot tasks, mAIHT is among the top two on most datasets and has the best mean accuracy.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use mAIHT instead of ADMM | LLaMA-7B pruning · WikiText2, sparsity 0.1 | perplexity: 5.6925 -> 5.6928 | Hurt |
| Use mAIHT instead of ADMM | LLaMA-7B pruning · WikiText2, sparsity 0.2 | perplexity: 5.8045 -> 5.8042 | Helped |
| Use mAIHT instead of ADMM | LLaMA-7B pruning · WikiText2, sparsity 0.3 | perplexity: 5.9586 -> 5.9565 | Helped |
| Use mAIHT instead of ADMM | LLaMA-7B pruning · WikiText2, sparsity 0.4 | perplexity: 6.3268 -> 6.3240 | Helped |
| Use mAIHT instead of ADMM | LLaMA-7B pruning · WikiText2, sparsity 0.5 | perplexity: 7.0826 -> 7.0720 | Helped |
| Add monotone acceleration to IHT (ablation) | IHT pruning of LLaMA-7B layers · Layer reconstruction | reconstruction error | Helped |
Surprising findings
- At 0.1 sparsity, mAIHT is slightly inferior to ADMM on perplexity.
Limitations
- Advantage is mainly at low-to-moderate sparsity; at 0.1 sparsity ADMM is slightly better.
- Theoretical risk bounds rely on RIP-type assumptions and bounded data distribution assumptions.
- Evaluated only on LLaMA-7B.
Experimental setup
Datasets: C4 WikiText2
Benchmarks: WikiText2 BoolQ RTE HellaSWAG WinoGrande ARC easy ARC challenge OpenbookQA
Metrics: perplexity accuracy reconstruction error
Baselines: Magnitude Pruning Wanda SparseGPT ADMM-based pruning
Builds on: Wanda SparseGPT ADMM-based pruning Iterative hard thresholding Monotone accelerated proximal gradient (Li & Lin 2015)
Closest papers by meaning
- SlimLLM: Accurate Structured Pruning for Large Language Models (ICML 2025 · similarity 0.93)
- SlimGPT: Layer-wise Structured Pruning for Large Language Models (NeurIPS 2024 · similarity 0.93)
- Pruner-Zero: Evolving Symbolic Pruning Metric From Scratch for Large Language Models (ICML 2024 · similarity 0.93)
- AlphaPruning: Using Heavy-Tailed Self Regularization Theory for Improved Layer-wise Pruning of Large Language Models (NeurIPS 2024 · similarity 0.93)
- Toward Adaptive Large Language Models Structured Pruning via Hybrid-grained Weight Importance Assessment (AAAI 2025 · similarity 0.92)
- Fluctuation-Based Adaptive Structured Pruning for Large Language Models (AAAI 2024 · similarity 0.92)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.