Research map / Large language models / Instruction and fine-tuning
Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Prunes LLaMA2-7B into 1.3B and 2.7B models via learned targeted structured pruning, then continues pre-training on 50B RedPajama tokens with dynamic batch loading that reweights domains by loss reduction rate.
Problem
Training small competitive LLMs from scratch is prohibitively expensive; structured pruning of general-purpose LLMs typically degrades performance and yields non-uniform architectures that are slow at inference.
Approach
Learn pruning masks (layers, heads, intermediate and hidden dims) with hard-concrete parameterization and Lagrange-constrained optimization to hit a specified uniform target architecture, then continue pre-training with dynamic batch loading that adjusts domain sampling weights so each domain's loss reaches its scaling-law reference loss at similar times.
Key results
- Sheared-LLaMA-1.3B (50B tokens) averages 51.0 on 11 downstream tasks, above TinyLlama-1.1B (50.0, trained on 3T tokens).
- Sheared-LLaMA-2.7B averages 56.7, above INCITE-Base-3B (54.7) and OpenLLaMA-3B-v2 (55.7) using about 3% of the compute.
- Dynamic batch loading shifts data use toward C4 (49.2% vs 15.0%) and Book, and balances loss differences across domains.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use dynamic batch loading instead of the original RedPajama domain proportions (ablation) | Sheared-LLaMA-1.3B · Downstream tasks | downstream performance | Helped |
| use scaling reference rather than source reference for dynamic batch loading (ablation) | Sheared-LLaMA-1.3B · Downstream tasks, especially math and coding | downstream performance | Helped |
| allocate more of a fixed 5B-token budget to pruning (ablation) | Sheared-LLaMA · Pruning and continued pre-training | perplexity: 12.99; 10.29; 9.01; 8.04 | Helped |
| use targeted structured pruning rather than CoFiPruning (ablation) | Sheared-LLaMA pruning · 1.3B and 2.7B pruning comparison | generation speed: 51 -> 58; 37 -> 43 | Helped |
| use targeted structured pruning rather than CoFiPruning (ablation) | Sheared-LLaMA pruning · 1.3B and 2.7B pruning comparison | validation perplexity: 9.1 -> 10.3; 7.0 -> 7.7 | Hurt |
| use targeted structured pruning rather than LLM-Pruner at similar parameter scale (ablation) | Sheared-LLaMA · LLM pruning comparison | perplexity: 7.09 -> 6.85 | Helped |
| use targeted structured pruning rather than LLM-Pruner at similar parameter scale (ablation) | Sheared-LLaMA · LLM pruning comparison | generation speed: 43 tokens/s -> 58 tokens/s | Helped |
| use targeted structured pruning rather than LLM-Pruner at similar parameter scale (ablation) | Sheared-LLaMA · LLM pruning comparison | training throughput: 83K tokens/s -> 139K tokens/s | Helped |
| prune Pythia-440M to 160M and continue with 10B tokens (ablation) | Sheared-Pythia · Pythia-160M zero-shot | performance: 43.56 -> 43.51 | No clear effect |
| prune Pythia-440M to 160M and continue with 33B tokens (ablation) | Sheared-Pythia · Pythia-160M zero-shot | performance: 43.56 -> 45.78 | Helped |
| exclude GitHub, StackExchange, and ArXiv data during pruning (ablation) | Pruned LLaMA1-13B to 7B · Domain loss after continued pre-training | loss disparity across domains | No clear effect |
| prune LLaMA1-13B to 7B rather than use LLaMA1-7B directly (ablation) | Pruned LLaMA1-13B · RedPajama domain losses | loss disparity across domains | Mixed |
Surprising findings
- Dynamic batch loading and pruning leave the GitHub loss below the reference while C4 lags, showing pruning retains knowledge unevenly across domains.
- Simply excluding easy domains during pruning does not fix the loss disparity after continued pre-training.
Limitations
- null
Experimental setup
Datasets: RedPajama ShareGPT
Benchmarks: SciQ PIQA WinoGrande ARC-E ARC-C HellaSwag LogiQA BoolQ LAMBADA NQ MMLU GSM8K
Metrics: accuracy exact match pass@1 win rate perplexity
Baselines: OPT-1.3B Pythia-1.4B TinyLlama-1.1B OPT-2.7B Pythia-2.8B INCITE-Base-3B OpenLLaMA-3B-v1 OpenLLaMA-3B-v2 CoFiPruning LLM-Pruner
Builds on: CoFiPruning LLaMA2 RedPajama Pythia DoReMi
Closest papers by meaning
- SlimLLM: Accurate Structured Pruning for Large Language Models (ICML 2025 · similarity 0.94)
- Toward Adaptive Large Language Models Structured Pruning via Hybrid-grained Weight Importance Assessment (AAAI 2025 · similarity 0.94)
- SlimGPT: Layer-wise Structured Pruning for Large Language Models (NeurIPS 2024 · similarity 0.93)
- Layer as Puzzle Pieces: Compressing Large Language Models through Layer Concatenation (NeurIPS 2025 · similarity 0.92)
- Pruning Large Language Models with Semi-Structural Adaptive Sparse Training (AAAI 2025 · similarity 0.92)
- An Efficient Pruner for Large Language Model with Theoretical Guarantee (ICML 2025 · similarity 0.92)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.