Research map / Large language models / Language, safety and interpretability
Probe Pruning: Accelerating LLMs through Dynamic Pruning via Model-Probing
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Probe Pruning, a training-free online dynamic structured pruning method that uses a small probe of each batch (5% samples, 50% tokens, ~1.5% FLOPs) to run a few layers ahead and select weight channels per batch, fused with historical states.
Problem
Structured pruning of LLMs without fine-tuning suffers large performance gaps because calibration-based pruning decisions are static and ignore batch-dependent activation outliers, while dynamic pruning needs intermediate hidden states that are not available before computation.
Approach
Select a small, high-residual-importance probe of samples and tokens from each batch, run it a few layers ahead to get intermediate states, fuse them with historical statistics via importance-scaled fusion, and prune channels per batch using a new squared-Wanda-based structured metric (PPsp).
Key results
- At 40% pruning on LLaMA-2-7B, PP reaches WikiText2 perplexity 16.8 vs 38.9 for FLAP and 43.8 for Wanda-sp (dense 6.0), without fine-tuning.
- PP's performance runtime ratio is 37.37 versus 95.64 for FLAP and 106.48 for Wanda-sp, a 2.56x improvement over FLAP.
- Probing costs about 1.5% of dense FLOPs (66 vs 4420 TFLOPs on WikiText2), and PP outperforms LoRAPrune and LLM-Pruner even when those are fine-tuned on LLaMA-2-13B at 40% (61.0 vs 55.5 and 54.7 average accuracy).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use dynamic Probe Pruning rather than calibration-based FLAP | LLaMA-2-7B at 40% pruning · WikiText2 | perplexity: 38.9(1.3) -> 16.8(0.1) | Helped |
| use dynamic Probe Pruning rather than FLAP | LLaMA-3-8B at 40% MLP pruning · Commonsense reasoning | average accuracy: 53.6 -> 58.9 | Helped |
| identify probe samples and tokens without residual importance (ablation) | Probe Pruning on LLaMA-2-7B at 40% pruning · WikiText2 | perplexity: 16.8(0.1) -> 37.1(0.4) | Hurt |
| identify probe samples and tokens without residual importance (ablation) | Probe Pruning on LLaMA-2-7B at 40% pruning · Commonsense reasoning | average accuracy: 56.6 -> 42.3 | Hurt |
| remove historical states from pruning decisions (ablation) | Probe Pruning on LLaMA-2-7B at 40% pruning · WikiText2 | perplexity: 16.9(0.1) -> 20.1(0.3) | Hurt |
| remove historical states from pruning decisions (ablation) | Probe Pruning on LLaMA-2-7B at 40% pruning · Commonsense reasoning | average accuracy: 51.2 -> 56.6 | Hurt |
| replace PPsp with Wanda-sp as the pruning metric (ablation) | LLaMA-2-7B fixed-pruned model at 40% pruning · WikiText2 | perplexity: 29.7(0.3) -> 43.8(1.5) | Hurt |
| replace PPsp with FLAP as the pruning metric (ablation) | OPT-13B fixed-pruned model at 40% pruning · WikiText2 | perplexity: 35.5(0.3) -> 41.1(0.3) | Hurt |
| use C4 rather than WikiText2 validation for calibration (ablation) | FLAP on LLaMA-2-7B at 40% pruning · WikiText2 | perplexity: 18.5(0.2) -> 38.9(1.3) | Hurt |
| square the FLAP attention metric (ablation) | FLAP on LLaMA-2-7B at 40% overall pruning · WikiText2 | perplexity: 38.9(1.3) -> 29.1(0.4) | Helped |
| increase probe batch size from 1 to 20 (ablation) | Probe Pruning at 40% pruning · WikiText2 | perplexity | Helped |
| run probing in parallel with computation of earlier pruned blocks (ablation) | PP-Parallel on LLaMA-2-7B at 40% pruning · WikiText2 | perplexity: 16.8(0.1) -> 17.9(0.1) | Hurt |
Surprising findings
- FLAP calibrated on WikiText2 validation reaches perplexity 18.5 at 40% pruning, versus 38.9 when calibrated on C4, showing calibration-data bias.
- Pruning attention heads saturates faster than MLP at high ratios, so attention and MLP ratios should be set independently.
Limitations
- MLP-block runtime speedup is smaller (1.30x) than baselines due to system overhead such as weight extraction.
- Parallel probing feasibility is only demonstrated, with hardware-dependent actual parallelism.
- Evaluation limited to zero-shot perplexity and commonsense tasks on a few model families; no generation or instruction-following evaluation.
Experimental setup
Datasets: C4 WikiText2 BoolQ PIQA HellaSwag WinoGrande ARC-Easy ARC-Challenge OpenbookQA
Benchmarks: WikiText2 BoolQ PIQA HellaSwag WinoGrande ARC-Easy ARC-Challenge OpenbookQA
Metrics: perplexity accuracy performance runtime ratio (PRR) Jaccard index FLOPs runtime speedup
Baselines: Wanda-sp FLAP LLM-Pruner LoRAPrune Full-Batch Probing
Builds on: Wanda FLAP LLM-Pruner LoRAPrune
Closest papers by meaning
- Fluctuation-Based Adaptive Structured Pruning for Large Language Models (AAAI 2024 · similarity 0.92)
- Týr-the-Pruner: Structural Pruning LLMs via Global Sparsity Distribution Optimization (NeurIPS 2025 · similarity 0.92)
- Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning (ICLR 2024 · similarity 0.92)
- Toward Adaptive Large Language Models Structured Pruning via Hybrid-grained Weight Importance Assessment (AAAI 2025 · similarity 0.91)
- Let LLM Tell What to Prune and How Much to Prune (ICML 2025 · similarity 0.91)
- SlimGPT: Layer-wise Structured Pruning for Large Language Models (NeurIPS 2024 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.