Research map / Large language models / Language, safety and interpretability
Stacking Your Transformers: A Closer Look at Model Growth for Efficient LLM Pre-Training
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Systematically evaluates four model growth operators for LLM pre-training and shows depthwise stacking (G_stack) accelerates training of Llama-like 1.1B to 7B models, with guidelines for growth timing and factor.
Problem
Model growth methods that initialize large models from smaller trained ones are underexplored for efficient decoder-only LLM pre-training, lacking comprehensive evaluation, scalability tests, and empirical guidelines.
Approach
Summarizes existing growth methods into four atomic operators (direct, learned, zero, random) in widthwise and depthwise forms, finds one-hop depthwise stacking (G_stack) best, and scales it to 7B and 750B tokens with fitted equations for growth timing and factor.
Key results
- G_stack 3B model achieves a 54.5% speedup over the baseline, with a 2.1 point average accuracy gain over the baseline trained on 240B tokens.
- Compared to a conventionally trained 7B model using 300B tokens, G_stack converges to the same loss with 194B tokens (54.6% speedup).
- At 1.1B with 100B tokens, G_stack increases training speed by 49.1% over training from scratch and reaches 52.80 average vs 50.09 for baseline on the 1B comparison.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use depthwise stacking instead of training from scratch | 1.1B LLM · ARC-c, ARC-e, PIQA, Sciq, Winogrande (100B tokens) | accuracy: ARC-c: 27.21 -> 29.18; ARC-e: 48.86 -> 54.25; PIQA: 69.64 -> 71.98; Sciq: 76.8 -> 81.1; Winogrande: 54.53 -> 56.03 | Helped |
| Use depthwise stacking instead of training from scratch | 1.1B LLM · Lambada and Logiqa (100B tokens) | accuracy: Lambada: 47.87 -> 48.20; Logiqa: 25.96 -> 28.87; Pythia-1B: 53.52 on Lambada and 29.49 on Logiqa | Mixed |
| Use depthwise stacking instead of training from scratch | 1.1B LLM · Average of seven benchmarks (100B tokens) | accuracy: 50.09 -> 52.80 | Helped |
| Use depthwise stacking instead of training from scratch | 3B LLM · Instruction-tuned evaluation | average accuracy: 55.8 -> 57.34 | Helped |
| Train with depthwise stacking rather than from scratch | 1.1B LLM pre-training · Training to the same loss | speedup: +49.1% | Helped |
| Replace one-hop stacking with gradual train-stack-train-stack growth (ablation) | G_stack, 1.1B LLM · 100B-token pre-training | average accuracy: 2.4 higher average accuracy for G_stack | Hurt |
| Replace one-hop stacking with gradual train-stack-train-stack growth (ablation) | G_stack, 1.1B LLM · 100B-token pre-training | Wikitext perplexity: 0.6 better Wikitext PPL for G_stack | Hurt |
| Add 20% noise to strict function-preserving G_direct width growth (ablation) | G_direct width growth · Wikitext after 40B tokens | perplexity: 0.27 | Helped |
| Add 20% noise to strict function-preserving G_direct width growth (ablation) | G_direct width growth · Average benchmark accuracy after 40B tokens | average accuracy: 0.41 | Helped |
| Add 20% noise to G_stack (ablation) | G_stack · LLM pre-training evaluation | performance | Helped |
| Increase noise beyond 20% in G_stack (ablation) | G_stack · LLM pre-training evaluation | performance | Hurt |
| Use a one-layer base model rather than a deeper base model for stacking (ablation) | G_stack, 3B LLM · IsoFLOP comparison | loss | Hurt |
Surprising findings
- Depthwise stacking, which violates function preservation, outperforms function-preserving growth operators.
- Adding 20% noise to the strict function-preserving direct widthwise growth improves over the original, suggesting function preservation is not the sole factor for growth success.
Limitations
- Limited compute restricted the growth-factor and growth-timing search, so the growth factor equation is not fitted and g=4 is a constant suggestion.
- Only simple one-hop operators are studied; multi-step growth and learning-rate dynamics are not explored.
- Cosine learning rate schedule was incomplete because training runs were interrupted; no theoretical explanation provided.
Experimental setup
Datasets: SlimPajama-627B
Benchmarks: ARC-c ARC-e LAMBADA LogiQA PIQA SciQ Winogrande WikiText perplexity Hellaswag TruthfulQA MMLU
Metrics: training loss accuracy perplexity speedup (FLOPs-based)
Baselines: training from scratch LiGO StackedBERT gradual stacking Pythia-1B TinyLlama-1.1B
Builds on: Net2Net StackedBERT LiGO Lemon TinyLlama
Closest papers by meaning
- Masked Structural Growth for 2x Faster Language Model Pre-training (ICLR 2024 · similarity 0.90)
- Unveiling the Secret Recipe: A Guide For Supervised Fine-Tuning Small LLMs (ICLR 2025 · similarity 0.89)
- Dynamic Sparse No Training: Training-Free Fine-tuning for Sparse LLMs (ICLR 2024 · similarity 0.89)
- Benchmarking Optimizers for Large Language Model Pretraining (NeurIPS 2026 · similarity 0.89)
- Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning (ICLR 2024 · similarity 0.88)
- Building on Efficient Foundations: Effective Training of LLMs with Structured Feedforward Layers (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.