Research map / Large language models / Language, safety and interpretability
DISP-LLM: Dimension-Independent Structural Pruning for Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DISP-LLM, a dimension-independent structural pruning method that removes the structural dependence along the embedding dimension using index selection/addition and hypernetwork-learned per-block widths, with no weight updates and no extra parameters.
Problem
Structural pruning of LLMs is constrained by residual connections that force all layers to share the same embedding-dimension subset and width, while prior flexible alternatives (SliceGPT) add substantial extra parameters.
Approach
Relocate the selection matrices inside the residual stream so each block selects its own subset of embedding features via index selection and index addition, leaving the residual connection unchanged; learn per-layer widths with a GRU-based hypernetwork trained through ReinMax straight-through gradients under a parameter-budget regularizer, without updating base weights.
Key results
- On WikiText-2 at 50% pruning, DISP-LLM reaches PPL 9.84 on LLaMA-2 7B and 7.11 on LLaMA-2 13B, beating LLM Surgeon (15.38 and 9.43) without weight updates.
- Against 2:4 semi-structured SparseGPT and Wanda, DISP-LLM (50% structural) matches or beats them on LLaMA-2 7B/13B and LLaMA 13B (8.15 vs 9.11 SparseGPT on LLaMA 13B).
- On LLaMA-2 7B at 50% pruning, DISP-LLM Alpaca averages 51.05 zero-shot accuracy versus 45.68 for LLM Surgeon; the paper reports DISP-LLM is 27.39x and 14.76x cheaper than LLM Surgeon on LLaMA-2 7B and 13B.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use dimension-independent pruning instead of LLM Surgeon at 50% pruning | LLaMA-2 7B · WikiText-2 | PPL: 15.38 -> 9.84 | Helped |
| Use dimension-independent pruning instead of LLM Surgeon at 50% pruning | LLaMA-2 13B · WikiText-2 | PPL: 9.43 -> 7.11 | Helped |
| Use dimension-independent pruning instead of SparseGPT 2:4 pruning | LLaMA 7B · WikiText-2 | PPL: 11.00 -> {11.47} | Hurt |
| Use dimension-independent pruning instead of SparseGPT 2:4 pruning | LLaMA 13B · WikiText-2 | PPL: 9.11 -> \textbf{8.15} | Helped |
| Train the pruning structure on Alpaca rather than WikiText | DISP-LLM, LLaMA-2 7B at 50% pruning · Zero-shot tasks | average task accuracy: 46.72 -> 51.05 | Helped |
| Use dimension-independent pruning instead of LLM Surgeon at 30% pruning | LLaMA-2 7B · Zero-shot tasks | average task accuracy: 59.03 -> 58.10 | Hurt |
| Use DISP-LLM instead of SliceGPT | Phi-2 at 30% pruning · Zero-shot tasks | average task accuracy: 51.99 -> 59.00 | Helped |
| Add a hypernetwork rather than using element-wise gates (ablation) | DISP-LLM on Phi-1.5 at 50% compression · WikiText-2 | PPL: 47.29 -> 32.89 | Helped |
| Keep Bi-GRU in the hypernetwork rather than removing it (ablation) | DISP-LLM on Phi-1.5 at 50% compression · WikiText-2 | PPL: 37.43 -> 32.89 | Helped |
| Break structural dependence rather than constrain embedding-dimension pruning (ablation) | DISP-LLM on Phi-1.5 · Zero-shot tasks | performance | Helped |
| Add LoRA fine-tuning after pruning | DISP-LLM Alpaca on LLaMA-2 7B at 50% pruning · Zero-shot tasks | average task accuracy: 51.05 -> 54.25 | Helped |
Surprising findings
- Structural pruning without weight updates matched or exceeded 2:4 semi-structured SparseGPT and Wanda on several models, which the authors describe as a first.
- SliceGPT's orthogonal projections add roughly 5% to 13% extra parameters to LLaMA-2 7B, so its actual pruning ratio is much lower than its nominal slicing ratio.
Limitations
- Index selection and addition add inference overhead, though throughput still improves in the reported experiments.
- Uses one selection matrix per block for QKV rather than separate ones, a design choice made for fair comparison and to limit indexing overhead.
Experimental setup
Datasets: WikiText-2 Alpaca PIQA WinoGrande HellaSwag ARC-e ARC-c
Benchmarks: WikiText-2 PIQA WinoGrande HellaSwag ARC-e ARC-c
Metrics: perplexity zero-shot accuracy throughput (tokens/s) cost ($)
Baselines: SliceGPT LLM-Pruner LLM Surgeon K-OBD SparseGPT Wanda Magnitude
Builds on: SliceGPT LLM-Pruner ReinMax Straight-Through estimator
Closest papers by meaning
- SlimLLM: Accurate Structured Pruning for Large Language Models (ICML 2025 · similarity 0.93)
- Layer as Puzzle Pieces: Compressing Large Language Models through Layer Concatenation (NeurIPS 2025 · similarity 0.92)
- AlphaPruning: Using Heavy-Tailed Self Regularization Theory for Improved Layer-wise Pruning of Large Language Models (NeurIPS 2024 · similarity 0.92)
- Týr-the-Pruner: Structural Pruning LLMs via Global Sparsity Distribution Optimization (NeurIPS 2025 · similarity 0.92)
- SlimGPT: Layer-wise Structured Pruning for Large Language Models (NeurIPS 2024 · similarity 0.92)
- Toward Adaptive Large Language Models Structured Pruning via Hybrid-grained Weight Importance Assessment (AAAI 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.