Research map / Architectures and efficiency / Pruning and efficient tuning
Numerical Pruning for Efficient Autoregressive Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a training-free structural pruning method for decoder-only autoregressive transformers (LLMs and LlamaGen image generators) that scores channels via a Newton's-method mask optimization and compensates pruned rows with a closed-form weight update.
Problem
Compressing decoder-only autoregressive transformers for both language and image generation while preserving performance, since existing structural pruning methods are mostly language-only and rely on costly fine-tuning or many calibration samples.
Approach
A numerical score derived from solving a relaxed mask optimization (minimizing layer output error under a pruning-ratio constraint) with Newton's method, used for global ranking of attention heads and MLP channels, plus a closed-form compensation update of remaining rows in output projections using 128 calibration samples.
Key results
- At 70% pruning on LLaMA-7B (WikiText2), the method reaches perplexity 162.9 versus 577.9 for FLAP and 3605 for SliceGPT, while LLM-Pruner gives 9010.
- On LlamaGen-3B at 384x384 with 20% pruning, FID improves from 162.15 (FLAP) to 20.16 with the proposed method.
- At 20% pruning on LLaMA-7B commonsense reasoning, average accuracy is 58.79 versus 58.58 for FLAP and 46.99 for SliceGPT.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use numerical-score pruning with compensation instead of FLAP pruning | LLaMA-7B · WikiText2, 2048 tokens, 10% pruning | perplexity: 6.34 -> 6.01 | Helped |
| Use numerical-score pruning with compensation instead of FLAP pruning | LLaMA-2-7B · WikiText2, 2048 tokens, 10% pruning | perplexity: 6.04 -> 5.88 | Helped |
| Use numerical-score pruning with compensation instead of FLAP pruning | LLaMA-3-8B · WikiText2, 2048 tokens, 10% pruning | perplexity: 7.40 -> 7.05 | Helped |
| Use numerical-score pruning with compensation instead of FLAP pruning | LLaMA-7B · common-sense reasoning, 20% pruning | Average Acc.: 58.58 -> 58.79 | Helped |
| Use numerical-score pruning with compensation instead of FLAP pruning | LlamaGen-XXL · ImageNet 384×384, 10% pruning | FID: 7.87 -> 6.09 | Helped |
| Use numerical-score pruning with compensation instead of FLAP pruning | LlamaGen-3B · ImageNet 384×384, 10% pruning | Inception Score: 158.74 -> 202.93 | Helped |
| Use numerical-score pruning with compensation instead of FLAP pruning | LlamaGen-3B · ImageNet 384×384, 20% pruning | FID: 162.15 -> 20.16 | Helped |
| Use numerical-score pruning with compensation instead of FLAP pruning | LLaMA-7B · WikiText2, 128 tokens, 10% pruning | perplexity: 13.84 -> 13.31 | Helped |
| Use numerical-score pruning with compensation instead of FLAP pruning | LLaMA-7B · WikiText2, 128 tokens, 20% pruning | perplexity: 14.62 -> 14.47 | Helped |
| Increase pruning ratio from 10% to 20% | Ours on LLaMA-7B · MMLU | accuracy: 31.8 -> 29.3 | Hurt |
| Increase pruning ratio from 10% to 20% | Ours on LLaMA-7B · GSM8K | accuracy: 4.3 -> 2.3 | Hurt |
| Use 128 compensation samples instead of 512 or 1024 (ablation) | Ours · compensation experiments across pruning ratios | perplexity | No clear effect |
Surprising findings
- LLM-Pruner and FLAP fail to produce usable models at 70% pruning on LLaMA-65B (NaN or very high perplexity).
Limitations
- Reduced effectiveness on smaller LlamaGen models for image generation because the discrete tokenizer loses detail as sparsity increases.
- Image generation evaluation limited to LlamaGen and compared mainly against FLAP.
Experimental setup
Datasets: WikiText2 ImageNet
Benchmarks: WikiText2 PTB C4 BoolQ PIQA HellaSwag WinoGrande ARC-easy ARC-challenge OpenbookQA MMLU GSM8K
Metrics: perplexity accuracy FID sFID IS Precision Recall GPU memory generation speed
Baselines: LLM-Pruner SliceGPT FLAP SparseGPT
Builds on: Optimal Brain Surgeon SparseGPT LLM-Pruner FLAP SliceGPT
Closest papers by meaning
- Computation and Memory-Efficient Model Compression with Gradient Reweighting (NeurIPS 2025 · similarity 0.89)
- A Fast Post-Training Pruning Framework for Transformers (NeurIPS 2022 · similarity 0.89)
- AutoCompress: An Automatic DNN Structured Pruning Framework for Ultra-High Compression Rates (AAAI 2020 · similarity 0.89)
- Resource-Efficient Transformer Pruning for Finetuning of Large Models (CVPR 2024 · similarity 0.88)
- Targeted Low-rank Refinement: Enhancing Sparse Language Models with Precision (ICML 2025 · similarity 0.87)
- OATS: Outlier-Aware Pruning Through Sparse and Low Rank Decomposition (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.