Research map / Large language models / Language, safety and interpretability

Let LLM Tell What to Prune and How Much to Prune

ICML 2025 · Large language models › Language, safety and interpretability · approach: LLM pruning and scaling data

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes a structured pruning framework that jointly prunes blocks, attention/MLP layers, and weight rows/columns for LLMs, using transfer entropy and information-entropy criteria to set dynamic per-unit pruning ratios without fine-tuning.

Problem

Existing structured pruning of LLMs targets a single structure unit (block, layer, or row/column) and assigns a prescribed uniform pruning ratio, which yields poor trade-offs between inference speed and performance.

Approach

Measures block and layer importance via transfer entropy from masking each module, allocates per-block pruning ratios dynamically (with DFS compensation for leftover budget), then distributes pruning within blocks to rows/columns via entropy-based K-means clustering, followed by weighted bias compensation.

Key results

• At 50% pruning, LLaMA2-7B reaches WikiText2 perplexity 20.07 versus 21.08 for SliceGPT and 58.86 for BlockPruner (dense 5.47).
• At 30% sparsity on LLaMA2-7B, average zero-shot accuracy is 56.65 versus 55.12 for FLAP and 54.77 for Wanda (dense 68.98).
• On LLaMA2-7B at 50% pruning, throughput is 38.78 tokens/s versus 34.38 for SliceGPT and 34.39 for FLAP (dense 29.86).

Measured effects reported in the paper

Surprising findings

• Semi-structured SparseGPT 2:4 degrades LLaMA3-8B perplexity to 19.99 (dense 5.53), worse than many structured methods at 50%.

Limitations

• Evaluation focuses mainly on perplexity and five zero-shot tasks; no reasoning or generation benchmarks reported.
• Relies on Gaussian assumption for entropy estimation and clustering heuristics (K-means, DFS) that are not thoroughly justified.
• Calibration limited to WikiText2/C4 with 128 samples.

Experimental setup

Datasets: WikiText2

Benchmarks: WikiText2 PIQA WinoGrande HellaSwag ARC-e ARC-c

Metrics: perplexity zero-shot accuracy tokens per second memory (MB)

Baselines: SparseGPT Wanda FLAP SliceGPT LLM-Pruner BlockPruner SLEB

Builds on: FLAP SliceGPT SLEB ShortGPT Wanda

Closest papers by meaning

• SlimLLM: Accurate Structured Pruning for Large Language Models (ICML 2025 · similarity 0.92)
• An Efficient Pruner for Large Language Model with Theoretical Guarantee (ICML 2025 · similarity 0.92)
• Toward Adaptive Large Language Models Structured Pruning via Hybrid-grained Weight Importance Assessment (AAAI 2025 · similarity 0.91)
• Probe Pruning: Accelerating LLMs through Dynamic Pruning via Model-Probing (ICLR 2025 · similarity 0.91)
• Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning (ICLR 2024 · similarity 0.91)
• SlimGPT: Layer-wise Structured Pruning for Large Language Models (NeurIPS 2024 · similarity 0.91)