Research map / Large language models / Language, safety and interpretability
Instruction-Following Pruning for Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes IFPruning, which uses a small sparsity predictor conditioned on the user instruction to select input-dependent FFN neurons of an LLM, jointly trained with the LLM via continued pre-training and SFT, so that a 9B model yields a 3B-active model.
Problem
Static structured pruning yields one fixed smaller model that cannot adapt to diverse tasks; contextual sparsity methods reload parameters at every decoding step and mostly target ReLU models.
Approach
A sparsity predictor (302M LM plus two-layer MLP head) maps the user prompt to FFN masks via SoftTopK; the masks select a fixed sub-network before decoding, and predictor and LLM are co-optimized with next-token loss on pre-training chunks and SFT data.
Key results
- The 9B-to-3B IFPruning model improves over the dense 3B model by about 8 points on coding and 5 on math, with marginal gap to the dense 9B.
- Per-task masks from a single task instruction retain performance close to per-input masks, with gaps mostly under 1%.
- Versus a dense 8B model pruned to 3B active, the IFPruning latency overhead for parameter selection and loading is under 0.1 s, and TTFT drops up to 57% and decode time up to 41%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use IFPruning 9B→3B instead of Dense-3B | IFPruning · HumanEval | pass@1: 35.2 -> 42.4 | Helped |
| Use IFPruning 9B→3B instead of Dense-3B | IFPruning · MBPP | pass@1: 28.8 -> 41.8 | Helped |
| Use IFPruning 9B→3B instead of Dense-3B | IFPruning · GSM8K | accuracy: 69.3 -> 72.0 | Helped |
| Use IFPruning 9B→3B instead of Dense-3B | IFPruning · MATH | accuracy: 31.8 -> 36.7 | Helped |
| Use IFPruning 9B→3B instead of Dense-3B | IFPruning · Tool Planning | performance: 25.2 -> 45.4 | Helped |
| Use IFPruning 9B→3B instead of Dense-3B | IFPruning · MMLU | accuracy: 61.8 -> 65.5 | Helped |
| Use IFPruning 9B→3B instead of Dense-3B | IFPruning · WinoGrande | accuracy: 69.8 -> 67.0 | Hurt |
| Add continued pre-training before SFT (ablation) | IFPruning 6B→3B · HumanEval | pass@1: 25.3 -> 31.9 | Helped |
| Add continued pre-training before SFT (ablation) | IFPruning 6B→3B · MBPP | pass@1: 24.4 -> 35.3 | Helped |
| Add continued pre-training before SFT (ablation) | IFPruning 6B→3B · MultiPL-E | pass@1: 15.3 -> 22.4 | Helped |
| Add continued pre-training before SFT (ablation) | IFPruning 6B→3B · GSM8K | accuracy: 50.9 -> 61.3 | Helped |
| Add continued pre-training before SFT (ablation) | IFPruning 6B→3B · MATH | accuracy: 13.5 -> 20.1 | Helped |
Surprising findings
- Per-task masks, selected from a single human-written task instruction with no extra training, sometimes match or beat per-input masks on coding and MMLU subsets.
- Continued pre-training before SFT gives large gains (e.g., HumanEval 25.3 to 31.9, MATH 13.5 to 20.1).
Limitations
- Only FFN dimensions are pruned; attention heads and hidden dimensions are left for future work.
- Relies on an internal proprietary SFT dataset and internal pre-trained models, limiting reproducibility.
- Batch serving with heterogeneous tasks is not addressed; per-request masks complicate server-side batching.
Experimental setup
Datasets: internal SFT dataset FLAN-V2 web-crawled pre-training data GPTeacher Code-Alpaca
Benchmarks: IFEval AlpacaEval 2.0 Arena-Hard-Auto HumanEval MBPP MultiPL-E GSM8K MATH ARC-Challenge ARC-Easy HellaSwag WinoGrande
Metrics: accuracy pass@1 length-controlled win rate win rate BLEU time-to-first-token decode time
Baselines: Dense-3B Pruning+Distill 3B Dense-9B Sheared LLaMA Qwen1.5-MoE-A2.7B
Builds on: SoftTopK Sheared LLaMA Tulu 2 Minitron
Closest papers by meaning
- The Journey Matters: Average Parameter Count over Pre-training Unifies Sparse and Dense Scaling Laws (ICLR 2025 · similarity 0.91)
- Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning (ICLR 2024 · similarity 0.91)
- Pruning Large Language Models with Semi-Structural Adaptive Sparse Training (AAAI 2025 · similarity 0.91)
- An Efficient Pruner for Large Language Model with Theoretical Guarantee (ICML 2025 · similarity 0.91)
- SlimLLM: Accurate Structured Pruning for Large Language Models (ICML 2025 · similarity 0.90)
- PAT: Pruning-Aware Tuning for Large Language Models (AAAI 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.