Research map / Large language models / Reasoning and chain of thought
SLEB: Streamlining LLMs through Redundancy Verification and Elimination of Transformer Blocks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SLEB, a training-free method that iteratively removes redundant transformer blocks in LLMs using a calibration-based perplexity metric, yielding real end-to-end inference speedups.
Problem
Pruning LLMs with unstructured, 2:4, or channel-wise methods rarely gives end-to-end speedup, and early-exit or layer-skipping approaches need dynamic decisions or training. The paper targets training-free, static, hardware-agnostic compression that yields speedup proportional to the pruning ratio.
Approach
Treats whole transformer blocks as the pruning unit, exploits high cosine similarity between neighboring block outputs, and greedily removes the block whose removal least increases calibration perplexity, recomputing scores after each removal.
Key results
- At 20% block sparsity on LLaMA-2-70B, SLEB reaches 1.26x latency speedup and 1.27x token-generation throughput with C4 perplexity 7.31 vs 5.71 dense.
- SLEB at 10% sparsity keeps LLaMA-2-70B zero-shot average at 73.14 vs 76.57 dense, outperforming SparseGPT (71.87) and Wanda (72.34) on average.
- Wanda collapses on OPT-66B (C4 perplexity 8414) and DSnoT fails similarly, while SLEB keeps perplexity at 11.25 at 10% sparsity.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Remove 10% of transformer blocks with SLEB | LLaMA-2-70B · C4 validation | perplexity: 5.71 -> 6.32 | Hurt |
| Remove 20% of transformer blocks with SLEB | LLaMA-2-70B · C4 validation | perplexity: 5.71 -> 7.31 | Hurt |
| Remove 10% of transformer blocks with SLEB | LLaMA-2-70B · Zero-shot task average | accuracy: 76.57 -> 73.14 | Hurt |
| Remove 20% of transformer blocks with SLEB | LLaMA-2-70B · Zero-shot task average | accuracy: 76.57 -> 70.81 | Hurt |
| Remove 10% of transformer blocks with SLEB | LLaMA-2-70B · Prompt processing, A100 | latency: 1718.4 -> 1529.1 | Helped |
| Remove 20% of transformer blocks with SLEB | LLaMA-2-70B · Prompt processing, A100 | latency: 1718.4 -> 1364.1 | Helped |
| Remove 10% of transformer blocks with SLEB | LLaMA-2-70B · Token generation, A100 | throughput: 299 -> 336 | Helped |
| Remove 20% of transformer blocks with SLEB | LLaMA-2-70B · Token generation, A100 | throughput: 299 -> 381 | Helped |
| Use iterative network-level block assessment (Metric³) instead of Metric¹ or Metric² to select blocks (ablation) | SLEB · OPT-6.7B and OPT-13B, WikiText-2 | perplexity | Helped |
| Select blocks with Metric¹ based on input-output distance (ablation) | SLEB · OPT-6.7B and OPT-13B, WikiText-2 | perplexity | Hurt |
| Use Metric² without recalculating block significance after each removal (ablation) | SLEB · OPT-6.7B, WikiText-2 | token prediction quality | Hurt |
| Increase SLEB block-removal sparsity from 10% to 20% (ablation) | OPT and LLaMA-2 models · C4 and zero-shot tasks | performance | Hurt |
Surprising findings
- Wanda and DSnoT completely fail on OPT-66B at 50% 2:4 sparsity, with perplexity in the thousands.
- Ideal early exit yields at most about 1.14x speedup at batch size 1 and essentially none beyond batch size 4, since about 90% of blocks still need processing.
Limitations
- Speedup and perplexity results are reported only for sparsity levels of 10% and 20%, with higher sparsity causing notable degradation.
- Removing whole blocks is coarse-grained, and the greedy search requires repeated forward passes, making pruning of 70B models take about 1.5 hours on 2 A100 GPUs.
- Evaluation is limited to OPT and LLaMA-2 families and to perplexity and zero-shot tasks.
Experimental setup
Datasets: WikiText-2 C4 PIQA WinoGrande HellaSwag ARC-easy ARC-challenge Alpaca
Benchmarks: WikiText-2 C4 PIQA WinoGrande HellaSwag ARC-easy ARC-challenge
Metrics: perplexity zero-shot accuracy throughput (tokens/s) latency (ms) speedup
Baselines: SparseGPT Wanda DSnoT LLM-Pruner SliceGPT
Builds on: SparseGPT Wanda SliceGPT LLM-Pruner early exit
Closest papers by meaning
- Let LLM Tell What to Prune and How Much to Prune (ICML 2025 · similarity 0.89)
- Streamlining Redundant Layers to Compress Large Language Models (ICLR 2025 · similarity 0.89)
- xLSTM 7B: A Recurrent LLM for Fast and Efficient Inference (ICML 2025 · similarity 0.89)
- Týr-the-Pruner: Structural Pruning LLMs via Global Sparsity Distribution Optimization (NeurIPS 2025 · similarity 0.89)
- La RoSA: Enhancing LLM Efficiency via Layerwise Rotated Sparse Activation (ICML 2025 · similarity 0.88)
- Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads (ICML 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.