Research map / Architectures and efficiency / Pruning and efficient tuning
DuoGPT: Training-free Dual Sparsity through Activation-aware Pruning in LLMs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DuoGPT, a training-free one-shot pruning method that combines unstructured weight pruning with runtime activation sparsity by extending OBC/SparseGPT with activation-aware calibration and dense-output residual correction.
Problem
Deploying LLMs during decoding is costly in memory and compute; existing one-shot pruning ignores runtime activation sparsity, and naively combining weight and activation sparsity degrades accuracy.
Approach
Reinterpret activation sparsity during single-batch decoding as dynamic structured weight sparsity, and extend OBC/SparseGPT calibration with sparse activations plus output residuals from the dense model (asymmetric calibration), with an efficient closed-form pruning score so 70B models calibrate on one A100.
Key results
- Under 50% dual sparsity, DuoGPT reaches 7.38 WikiText2 PPL on LLaMA-3-70B versus 7.54 for SparseGPT, with 72.56 average zero-shot accuracy.
- Against structured pruning at 1.39x speedup on LLaMA-2-7B, DuoGPT averages 60.52 versus 57.34 for the best structured baseline (2SSP), a gain of about 3 points.
- Calibrating LLaMA-3-70B takes roughly 2.3 GPU hours on a single A100 with offloading.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use DuoGPT instead of SparseGPT at 50% dual-sparsity | LLaMA-3-70B · WikiText2 | perplexity: 7.54 -> 7.38 | Helped |
| use DuoGPT instead of SparseGPT at 50% dual-sparsity | LLaMA-3-8B · zero-shot average | accuracy: 59.51 -> 60.04 | Helped |
| use DuoGPT instead of SparseGPT at 50% dual-sparsity | LLaMA-2-7B · WikiText2 | perplexity: 8.98 -> 8.58 | Helped |
| use DuoGPT instead of SparseGPT at 50% dual-sparsity | LLaMA-2-13B · zero-shot average | accuracy: 64.50 -> 64.88 | Helped |
| use DuoGPT instead of the strongest structured baseline | LLaMA-2-7B · zero-shot average | accuracy: 57.34 -> 60.52 | Helped |
| use DuoGPT instead of SparseGPT at matched SRAM weight fetches | LLaMA-3-8B · GSM8K, 5-shot | accuracy: 36.39 -> 36.77 | Helped |
| increase dual-sparsity from 30% to 65% (ablation) | DuoGPT on LLaMA-2-7B · WikiText PPL | perplexity: 5.84 -> 77.3 | Hurt |
| increase dual-sparsity from 30% to 40% (ablation) | DuoGPT on LLaMA-2-7B · WikiText PPL | perplexity: 5.84 -> 6.48 | Hurt |
| apply TEAL threshold-based activation pruning to DuoGPT (ablation) | DuoGPT 50% on LLaMA-2-7B · OBQA | accuracy: 39.00 -> 38.40 | Hurt |
| apply TEAL threshold-based activation pruning to DuoGPT (ablation) | DuoGPT 50% on LLaMA-2-7B · BoolQ | accuracy: 72.48 -> 71.77 | Hurt |
| apply TEAL threshold-based activation pruning to DuoGPT (ablation) | DuoGPT 50% on LLaMA-2-7B · Arc-C | accuracy: 35.67 -> 36.77 | Helped |
| quantize DuoGPT weights from FP16 to INT8 (ablation) | DuoGPT 50% dual-sparse LLaMA-2-7B · WikiText2 | perplexity: 8.58 -> 8.58 | No clear effect |
Surprising findings
- Wanda without activation-aware calibration degrades exponentially beyond 60% dual sparsity, while DuoGPT stays usable at higher levels.
Limitations
- No full spMspV GPU kernel; speedups reflect only activation-sparsity (spVM) kernel, a lower bound.
- Dual sparsity applied only to linear layers, not attention/KV cache.
- Theoretical bound relies on approximations (e.g., constant sigma_r and weight norm bound).
Experimental setup
Datasets: C4 WikiText2 PIQA HellaSwag WinoGrande ARC-Easy ARC-Challenge OpenBookQA BoolQ GSM8K
Benchmarks: WikiText2 PIQA HellaSwag WinoGrande ARC-Easy ARC-Challenge OpenBookQA BoolQ GSM8K
Metrics: perplexity zero-shot accuracy average accuracy end-to-end speedup model size
Baselines: SparseGPT Wanda ShortGPT 2SSP BlockPruner SliceGPT TEAL R-Sparse
Builds on: Optimal Brain Compression SparseGPT Wanda asymmetric calibration
Closest papers by meaning
- R-Sparse: Rank-Aware Activation Sparsity for Efficient LLM Inference (ICLR 2025 · similarity 0.91)
- La RoSA: Enhancing LLM Efficiency via Layerwise Rotated Sparse Activation (ICML 2025 · similarity 0.90)
- Pruning Large Language Models with Semi-Structural Adaptive Sparse Training (AAAI 2025 · similarity 0.90)
- Dynamic Sparse No Training: Training-Free Fine-tuning for Sparse LLMs (ICLR 2024 · similarity 0.90)
- The Sparsity Whisperer (NeurIPS 2026 · similarity 0.89)
- SoLA: Leveraging Soft Activation Sparsity and Low-Rank Decomposition for Large Language Model Compression (AAAI 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.