Research map / Architectures and efficiency / Pruning and efficient tuning
Prompt-based Depth Pruning of Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PuDDing, a prompt-routed dynamic depth pruning method that uses a BERT-based router to pick one of a small set of task-derived omission sets per prompt, pruning blocks of LLMs for faster on-device inference.
Problem
Static depth pruning removes the same transformer blocks regardless of input, but block importance is task-dependent, so one fixed omission set degrades some tasks. Dynamic per-token routing needs all weights in fast memory, which is unsuitable for memory-constrained on-device inference.
Approach
Route once per prompt: build a small pool of omission sets from multiple task calibration splits using a task likelihood loss (and its difference variant), then train a BERT-based router with MSE on soft labels to pick the minimum-loss omission set from the prompt, so only the chosen blocks are loaded.
Key results
- At 20% sparsity (7 of 32 blocks pruned) on LLaMA-3.1-8B, PuDDing reaches 61.93% average zero-shot accuracy vs. 57.24% for SLEB and 55.77% for Shortened LLaMA (dense 74.90%).
- PuDDing achieves 1.19-1.23x pre-fill speedup and 1.22-1.25x end-to-end speedup on A100 and RTX 6000 Ada, with router overhead of 4-8 ms.
- With LoRA fine-tuning at 20% sparsity, PuDDing reaches 68.01% average vs. 67.17% for SLEB + LoRA.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| route prompts to task-appropriate omission sets instead of using static pruning | LLaMA-3.1-8B with 7 blocks pruned · Commonsense reasoning | average accuracy (%): 57.24 -> 61.93 | Helped |
| use 10 candidate omission sets instead of 5 (ablation) | PuDDing · Zero-shot commonsense reasoning | average accuracy (%): 61.19 -> 61.93 | Helped |
| increase candidate omission sets from 10 to 30 (ablation) | PuDDing · Zero-shot commonsense reasoning | average accuracy (%): 61.93 -> 61.69 | Hurt |
| use task likelihood instead of batch-PPL for static omission selection (ablation) | Static pruning · Zero-shot commonsense reasoning | average accuracy (%): 59.32 -> 61.02 | Helped |
| use task likelihood instead of batch-PPL for dynamic routing (ablation) | Dynamic pruning with a trained router · Zero-shot commonsense reasoning | average accuracy (%): 58.19 -> 61.93 | Helped |
| train with log-likelihood labels and MSE instead of log-likelihood labels and cross-entropy (ablation) | PuDDing router · Zero-shot commonsense reasoning | average accuracy (%): 60.44 -> 61.93 | Helped |
| add diverse MathQA, PubMedQA, and SciQ calibration data (ablation) | PuDDing · Mixed commonsense and domain-specific tasks | average accuracy (%): 61.28 -> 62.37 | Helped |
| quantize weights to 8-bit with AWQ (ablation) | PuDDing · Zero-shot commonsense reasoning | average accuracy (%): 61.93 -> 61.68 | No clear effect |
| quantize weights to 4-bit with AWQ (ablation) | PuDDing · Zero-shot commonsense reasoning | average accuracy (%): 61.93 -> 58.58 | Hurt |
| reformat WinoGrande as question-answer and score the answer with task likelihood instead of sentence-level likelihood (ablation) | PuDDing · WinoGrande | accuracy (%): 61.09% -> 64.16% | Helped |
| replace block 30 pruning with block 29 pruning | SLEB omission set on LLaMA-3.1-8B · BoolQ | accuracy (%): 62.2% -> 38.0%, 62.5% -> 37.9% | Hurt |
| replace block 30 pruning with block 29 pruning | SLEB omission set on LLaMA-3.1-8B · PIQA and WinoGrande | accuracy | Mixed |
Surprising findings
- Naive SLEB per prompt (pruning using each prompt as calibration) performs worse than static SLEB (52.58% vs 57.24%).
- Pruning block 29 instead of block 30 in LLaMA-3.1-8B drops BoolQ accuracy by over 20 points while PIQA and WinoGrande are unchanged or slightly improved, showing task-dependent block importance.
Limitations
- Assumes access to labeled task datasets for calibration and router training; does not handle open-ended generation tasks.
- Uses a fixed number of pruned blocks for all prompts, not adapting depth to task difficulty.
- Router must load additional weights when prompts map to different omission sets, though overlap reduces this cost.
Experimental setup
Datasets: ARC-Challenge ARC-Easy HellaSwag PIQA WinoGrande WikiText-2 BookCorpus Alpaca MathQA PubMedQA SciQ
Benchmarks: ARC-Challenge ARC-Easy BoolQ HellaSwag PIQA WinoGrande OpenBookQA MathQA MMLU PubMedQA SciQ
Metrics: zero-shot accuracy average accuracy wall-clock latency (TTFT, pre-fill + generation) speedup
Baselines: SLEB Shortened LLaMA FLAP SliceGPT LLM-Streamline SLEB per prompt
Builds on: SLEB Shortened LLaMA D-LLM Mixture-of-Depths
Closest papers by meaning
- An Efficient Pruner for Large Language Model with Theoretical Guarantee (ICML 2025 · similarity 0.89)
- Instruction-Following Pruning for Large Language Models (ICML 2025 · similarity 0.88)
- SlimGPT: Layer-wise Structured Pruning for Large Language Models (NeurIPS 2024 · similarity 0.88)
- Beyond Linear Approximations: A Novel Pruning Approach for Attention Matrix (ICLR 2025 · similarity 0.88)
- EfficientLLaVA: Generalizable Auto-Pruning for Large Vision-language Models (CVPR 2025 · similarity 0.88)
- Resource-Efficient Transformer Pruning for Finetuning of Large Models (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.