Research map / Architectures and efficiency / RNNs and normalization
Preparing Lessons for Progressive Training on Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Apollo, a progressive-from-scratch training method that samples depths with low-value-prioritized sampling (LVPS), shares weights across layers, and expands depth by interpolation, reaching large FLOPs savings on BERT and GPT without a pretrained model.
Problem
Training Transformers from scratch is expensive, and existing progressive layer-stacking methods give little acceleration and suffer training instability, while methods reusing pretrained small models may not apply to new architectures.
Approach
Shared low-layer weights are trained on randomly sampled depths (biased toward shallow depths via LVPS) so they learn high-layer functionality before expansion, and depth is extended by interpolation rather than stacking to keep training stable.
Key results
- Apollo achieves 41.6% FLOPs saving and 41.1% wall-time saving for BERT-Base, versus 35.6% for bert2BERT and 29.5% for StackBERT.
- On GPT-Base, Apollo reaches a 47.9% acceleration ratio in FLOPs.
- Apollo's downstream GLUE average (85.02) and SQuAD average (84.15) are comparable to the scratch baseline and to pretrained-model methods bert2BERT and LiGO.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Apollo progressive training from scratch instead of Scratch training | BERT-Base · BERT-Base training | FLOPs saving: 41.6% | Helped |
| Use LVPS instead of no sampling (ablation) | Apollo · BERT-Base training | acceleration ratio: +9.3% | Helped |
| Use LVPS instead of Edge Sampling or Full Sampling (ablation) | Apollo · BERT-Base training | acceleration ratio: 41.6%; 4.1%; +17.1% | Helped |
| Use interpolation rather than stacking for layer expansion (ablation) | Apollo · BERT-Base training | acceleration ratio: 39.7% -> 41.6% | Helped |
| Use interpolation rather than stacking to expand BERT-Base/2 from 6 to 12 layers (ablation) | BERT-Base/2 · BERT-Base/2 expanded to BERT-Base | loss: 7.32 -> 3.97 | Helped |
| Use interpolation rather than stacking to expand BERT-Base/2 from 6 to 12 layers (ablation) | BERT-Base/2 · BERT-Base/2 expanded to BERT-Base | gradient (1e-3): 55.28$\pm$241.52 -> 44.01$\pm$225.76 | Helped |
| Use interpolation rather than stacking for Apollo layer expansion (ablation) | Apollo · BERT-Base training | gradient (1e-3), 12-layer stage: 1.42$\pm$3.50 -> 1.39$\pm$3.01 | Helped |
| Train Apollo with StackBERT-style progressive expansion instead | BERT-Base · BERT-Base training | FLOPs saving: 29.5% -> 41.6% | Helped |
Surprising findings
- Apollo, trained from scratch, outperforms methods that rely on a pretrained small model (bert2BERT, LiGO) in acceleration ratio.
- Directly stacking layers raises loss and gradients sharply, while interpolation keeps them lower, though both exceed a random model's gradient level.
Limitations
- Experiments limited to BERT and GPT encoder/decoder models at base scale on Wikipedia and BookCorpus.
- Hyperparameters (k, stage schedule) chosen by hand; sensitivity only partly explored.
Experimental setup
Datasets: English Wikipedia Toronto BookCorpus
Benchmarks: GLUE SQuAD v1.1 SQuAD v2.0
Metrics: FLOPs saving wall time saving F1 accuracy Matthews correlation GLUE average
Baselines: Scratch StackBERT bert2BERT LiGO
Builds on: StackBERT bert2BERT LiGO Net2Net
Closest papers by meaning
- Progressive Residual Warmup for Language Model Pretraining (NeurIPS 2026 · similarity 0.88)
- The Sharpness Disparity Principle in Transformers for Accelerating Language Model Pre-Training (ICML 2025 · similarity 0.88)
- Skip a Layer or Loop It? Learning Program-of-Layers in LLMs (ICML 2026 · similarity 0.87)
- A Multi-Level Framework for Accelerating Training Transformer Models (ICLR 2024 · similarity 0.87)
- The Curse of Depth in Large Language Models (NeurIPS 2025 · similarity 0.87)
- GPAS: Accelerating Convergence of LLM Pretraining via Gradient-Preserving Activation Scaling (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.