Research map / Large language models / Reasoning and chain of thought
Compact Language Models via Pruning and Knowledge Distillation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that structured pruning of Nemotron-4 15B (width, depth, with activation-based importance) followed by knowledge-distillation retraining on ~94B tokens yields compact 8B and 4B LLMs (Minitron) that match community models at 40x fewer tokens, and distills best-practice guidelines.
Problem
Training each LLM size in a deployment family from scratch is extremely compute- and data-intensive; it is unclear how to prune an existing LLM along multiple axes and recover accuracy with little retraining data.
Approach
A purely activation-based (forward-pass only) importance estimation across depth, neuron, attention head, and embedding axes, combined with lightweight search over pruned architectures and logit-based knowledge distillation from the unpruned teacher as the retraining method, with iterative prune-and-distill across model sizes.
Key results
- Minitron 8B uses 94B training tokens (40x fewer than Nemotron-3 8B's 3.8T) and achieves MMLU 63.8 vs 54.7 for Nemotron-3 8B and 65.3 for Llama-3 8B.
- Iterative prune-and-distill from 15B to 8B to 4B gives a 12% MMLU improvement over one-shot pruning to 4B (4B-Pruned-Distill from 8B: MMLU 42.45 vs 37.81 from 15B).
- Distillation-based retraining of pruned 4B model improves MMLU by 13.5 points over training a random-init 4B model with equivalent compute (37.81 vs 24.36 at iso-compute setting).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use width-only pruning rather than depth-only pruning (ablation) | Nemotron-4 15B compressed to about 8B · 8T validation | LM loss: 2.155 -> 2.049 | Helped |
| Combine depth pruning with width pruning instead of width-only pruning (ablation) | Nemotron-4 15B compressed to about 8B · 8T validation | LM loss: 2.049 -> 2.062 | Hurt |
| Use iterative embedding-dimension pruning rather than one-shot pruning (ablation) | Nemotron-4 15B · LM validation | LM loss: 1.92 -> 1.92 | No clear effect |
| Use logits-only KLD rather than CLM plus logits KLD (ablation) | Nemotron-3 8B retraining · Nemotron-3 8B validation | LM loss: 2.117 -> 2.107 | Helped |
| Use forward KLD rather than reverse KLD (ablation) | Minitron 8B depth-pruned · 8T validation | LM loss: 2.665 -> 2.155 | Helped |
| Add intermediate-state loss to logits-only distillation (ablation) | Minitron 8B · Minitron 8B | MMLU: 58.3 -> 58.0 | No clear effect |
| Prune Nemotron-4 15B and distill rather than conventionally retraining the pruned model (ablation) | 4B model · MMLU | accuracy: 24.57 -> 37.81 | Helped |
| Prune Nemotron-4 15B and distill rather than training from random initialization (ablation) | 4B model · HellaSwag | accuracy: 46.22 -> 51.04 | Helped |
| Distill a 4B model from Minitron 8B rather than directly from Nemotron-4 15B (ablation) | 4B model · MMLU | accuracy: 37.81 -> 42.45 | Helped |
| Distill with logits-only loss rather than logits plus intermediate-state loss (ablation) | Minitron 8B · HellaSwag | accuracy: 73.6 -> 73.8 | No clear effect |
| Retrain from the Phase 2 checkpoint using Phase 2 data only rather than Phase 1 plus Phase 2 data (ablation) | Minitron 8B width-pruned · MMLU | accuracy: 54.7 -> 61.9 | Helped |
| Use Minitron 8B rather than one-shot pruning Nemotron-4 15B as the starting point for 4B distillation (ablation) | 4B model · HellaSwag | accuracy: 51.04 -> 52.04 | Helped |
Surprising findings
- Width pruning outperformed depth pruning after retraining, even though depth-width variant had lower loss immediately post-pruning and rankings flip after about 200 steps.
- Logit-only KL distillation performed better than combining with cross-entropy loss or intermediate state losses when depth was not significantly pruned.
Limitations
- Pruning was only evaluated on Nemotron-4 15B and models up to ~15B; generalization to other architectures or scales is not established.
- Multiple comparisons rely on evaluations run by the authors, with some numbers copied from other papers.
Experimental setup
Datasets: Nemotron-4 8T pretraining blend Nemotron-4 continued training (CT) data blend WikiText2
Benchmarks: MMLU HumanEval MBPP ARC-Challenge HellaSwag TruthfulQA WinoGrande XL-Sum GSM8K MT-Bench IFEval ChatRAG-Bench
Metrics: accuracy pass@1 rougeL LM validation loss MT-Bench score prompt-level accuracy
Baselines: LLM-Pruner SliceGPT LaCo ShortGPT Sheared LLaMa Llama-3 8B Mistral 7B Gemma 7B Llama-2 7B Nemotron-3 8B Phi-2 Gemma2
Builds on: Nemotron-4 15B Megatron-LM ShortGPT block importance Gromov et al. depth pruning Sheared LLaMA
Closest papers by meaning
- Pruning Large Language Models with Semi-Structural Adaptive Sparse Training (AAAI 2025 · similarity 0.93)
- You Only Prune Once: Designing Calibration-Free Model Compression With Policy Learning (ICLR 2025 · similarity 0.93)
- Layer as Puzzle Pieces: Compressing Large Language Models through Layer Concatenation (NeurIPS 2025 · similarity 0.92)
- The LLM Surgeon (ICLR 2024 · similarity 0.92)
- Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning (ICLR 2024 · similarity 0.92)
- Efficient Hybrid Language Model Compression through Group-Aware SSM Pruning (NeurIPS 2025 · similarity 0.92)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.