Research map / Architectures and efficiency / Pruning and efficient tuning
Týr-the-Pruner: Structural Pruning LLMs via Global Sparsity Distribution Optimization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Týr-the-Pruner searches for a global non-uniform sparsity distribution over a supernet of locally pruned LLM layers, using expectation error accumulation and an iterative coarse-to-fine prune-and-search with evolutionary search.
Problem
Structural pruning of LLMs loses accuracy at high sparsity; local methods ignore global topology while global methods rank substructures independently, ignoring inter-structure dependencies and lacking end-to-end optimization.
Approach
Build a supernet by applying Hessian/gradient-informed local pruning at multiple sparsity ratios per layer, combine pruned copies with expectation error accumulation, and find the global sparsity distribution via evolutionary search on a distillation-style similarity objective, refining the sparsity interval iteratively.
Key results
- On Llama-3.1-70B at 50% sparsity, Týr-the-Pruner keeps 97% of dense average accuracy (68.56 vs 70.63), while the best baseline (ZipLM/OSSCAR) retains about 89%.
- On Llama-3.1-8B at 50% sparsity, perplexity 30.89 and average accuracy 47.79 versus 43.58 for search-only and 40.74 for isotropic pruning.
- Inference on MI250: 50% sparsity reduces Llama-3.1-8B TTFT by 1.75x and raises decode throughput by 1.38x.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply Týr-the-Pruner instead of FLAP at 37.5% sparsity | Llama-3.1-8B · WikiText2 test, 37.5% sparsity | perplexity: 21.54 -> 18.09 | Helped |
| Apply Týr-the-Pruner instead of FLAP at 37.5% sparsity | Llama-3.1-8B · Average downstream accuracy, 37.5% sparsity | accuracy: 43.07 -> 53.46 | Helped |
| Prune with Týr-the-Pruner at 50% sparsity | Llama-3.1-70B · Average downstream accuracy | accuracy: 70.63 -> 68.56 | Hurt |
| Add progressive pruning (ablation) | Local pruning on Llama-3.1-8B · 50% sparsity | perplexity: 63.48 -> 58.09 | Helped |
| Add second-order Hessian information (ablation) | Local pruning on Llama-3.1-8B · 50% sparsity | perplexity: 109.88 -> 58.09 | Helped |
| Add first-order gradient information (ablation) | Local pruning on Llama-3.1-8B · 50% sparsity | perplexity: 67.31 -> 58.09 | Helped |
| Use FineWeb-Edu instead of Wikitext2 for calibration (ablation) | Local pruning on Llama-3.1-8B · ARC-Challenge, 50% sparsity | accuracy: 20.05 -> 24.06 | Helped |
| Use FineWeb-Edu instead of C4 for calibration (ablation) | Local pruning on Llama-3.1-8B · WikiText2, 50% sparsity | perplexity: 73.07 -> 58.09 | Helped |
| Add expectation error accumulation (ablation) | Local pruning and supernet construction on Llama-3.1-8B · WikiText2, 50% sparsity | perplexity: 538.23 -> 66.38 | Helped |
| Use expectation error accumulation instead of random error accumulation (ablation) | Supernet construction on Llama-3.1-8B · WikiText2, 50% sparsity | perplexity: 208.92 -> 66.38 | Helped |
| Use expectation error accumulation instead of uniform error accumulation (ablation) | Supernet construction on Llama-3.1-8B · WikiText2, 50% sparsity | perplexity: 75.10 -> 66.38 | Helped |
| Use similarity-based logits-only search instead of WikiText2 perplexity search (ablation) | Search on Llama-3.1-8B · ARC-Challenge, 50% sparsity | accuracy: 29.69 -> 31.83 | Helped |
Surprising findings
- Searched sparsity distributions do not follow simple depth-based priors: the 2nd FFN layer is largely retained while the 12th is fully pruned.
- Single-task (WikiText-2 perplexity) search search direction yields lower perplexity but worse downstream accuracy than the similarity-based metric.
Limitations
- Search time cost remains non-negligible despite reduced search space.
- Requires large disk storage for supernet substructures (e.g., 414.7GB for 70B).
Experimental setup
Datasets: FineWeb-Edu WikiText-2 ARC-Easy ARC-Challenge BoolQ HellaSwag OpenBookQA RTE WinoGrande MMLU Alpaca-GPT4
Benchmarks: WikiText-2 ARC-Easy ARC-Challenge BoolQ HellaSwag OpenBookQA RTE WinoGrande MMLU
Metrics: perplexity accuracy average accuracy retention
Baselines: ShortGPT LaCO+ SliceGPT Wanda-sp LLM-Pruner ZipLM OSSCAR FLAP SearchLLM DarwinLLM DISP-LLM ProbePruning
Builds on: OBS (Optimal Brain Surgeon) ZipLM EvoPress FLAP LLM-Pruner
Closest papers by meaning
- SlimGPT: Layer-wise Structured Pruning for Large Language Models (NeurIPS 2024 · similarity 0.93)
- Discovering Sparsity Allocation for Layer-wise Pruning of Large Language Models (NeurIPS 2024 · similarity 0.92)
- Fluctuation-Based Adaptive Structured Pruning for Large Language Models (AAAI 2024 · similarity 0.92)
- Layer as Puzzle Pieces: Compressing Large Language Models through Layer Concatenation (NeurIPS 2025 · similarity 0.92)
- AlphaPruning: Using Heavy-Tailed Self Regularization Theory for Improved Layer-wise Pruning of Large Language Models (NeurIPS 2024 · similarity 0.92)
- SparseLLM: Towards Global Pruning of Pre-trained Language Models (NeurIPS 2024 · similarity 0.92)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.