Research map / Architectures and efficiency / RNNs and normalization
Efficient stagewise pretraining via progressive subnetworks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes progressive subnetwork training (instantiated as RaPTr), which trains random depth-wise subnetworks whose size grows over stages, showing dropping can match or beat gradual stacking for BERT and UL2 pretraining.
Problem
Stagewise pretraining for efficiency has been dominated by layer stacking; layer dropping (e.g., progressive layer dropping) is considered ineffective, and stacking is sensitive to schedules and lacks anytime full-model evaluation.
Approach
Train random subnetworks (layer-skipping or reduced-width) within a shared full model, progressively increasing subnetwork size across stages (simple-to-complex), so every parameter contributes throughout and the full model is tracked; theoretically argues loss stability at stage transitions depends on residual connections and layer norm.
Key results
- On BERT-Base, RaPTr (6-8-10-12) reaches eval loss 1.75 vs 1.76 for baseline at 1.33x fewer FLOPs, and gets 88.1 average GLUE-subset vs 87.8 baseline and 87.3 gradual stacking.
- On UL2-1.6B at 1.2x fewer FLOPs, RaPTr matches baseline eval loss (2.06) and improves 1-shot SuperGLUE-style average from 40.4 (baseline) to 41.9, with TydiQA +2.3 and SQuAD v2 +2.0 over baseline.
- Wall-clock: UL2 1.2x FLOP reduction gives 1.19x speedup; BERT 1.33x FLOP reduction gives 1.26x speedup.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| train with progressively larger random layer subnetworks rather than full-model baseline training | RaPTr on BERT-Base · BERT-Base | Eval loss: 1.76 -> 1.75 | Helped |
| train with progressively larger random layer subnetworks rather than full-model baseline training | RaPTr on UL2-1.6B · UL2-1.6B | Avg.: 40.4 -> 41.9 | Helped |
| use progressively larger random layer subnetworks instead of progressively dropping more layers | RaPTr vs PLD · UL2-1.6B | Avg.: 38.5 -> 41.9 | Helped |
| use progressively larger random layer subnetworks instead of gradual stacking | RaPTr vs Stacking 6-9-12 · BERT-Base | Eval loss: 1.77 -> 1.75 | Helped |
| use progressive random subnetworks rather than the corresponding MSG growth operator | Attention-RaPTr · BERT-Base | Eval Loss: 2.07 -> 1.84 | Helped |
| use progressive random subnetworks rather than the corresponding MSG growth operator | MLP-RaPTr · BERT-Base | Eval Loss: 1.88 -> 1.84 | Helped |
| use schedule 6-8-10-12 instead of schedule 3-6-9-12 (ablation) | RaPTr on BERT-Base · BERT-Base | Avg.: 87.9 -> 88.1 | No clear effect |
| fix the first and last layers throughout training rather than use no fixed layers (ablation) | RaPTr on BERT-Base · BERT-Base | Eval loss: 2.12 -> 2.11 | Helped |
| add 30k initial full-model training (ablation) | RaPTr 12-16-20-24 on UL2-1.6B · UL2-1.6B | Avg.: 40.3 -> 41.9 | Helped |
| change schedule from 12-16-20-24 to 6-12-18-24 (ablation) | RaPTr with 30k initial full-model training on UL2-1.6B · UL2-1.6B | Avg.: 41.9 -> 41.2 | No clear effect |
Surprising findings
- Layer-dropping with increasing subnetwork size (RaPTr) outperforms progressive layer dropping, which drops more layers over time and performs significantly worse.
- Training loss decreases smoothly across RaPTr stage transitions without explicit loss-preservation mechanisms.
Limitations
- Schedule selection is not well understood and requires expensive compute to tune.
- Theory does not explain observed decrease in loss during transitions or the role of initial full-model warmup for UL2.
- Analysis of stability is limited to linear residual networks and informal lemmas; only single-axis subnetworks (depth, MLP, heads) evaluated.
Experimental setup
Datasets: Wikipedia BooksCorpus C4 Arxiv GitHub MNLI QNLI SST-2 TriviaQA TydiQA SQuAD v2 SuperGLUE
Benchmarks: GLUE MNLI GLUE QNLI GLUE SST-2 TriviaQA TydiQA SQuAD v2 SuperGLUE Natural QA Web QA DROP CoQA QuAC
Metrics: evaluation loss / perplexity accuracy exact match F1 relative FLOPs wall-clock speedup
Baselines: Baseline full-model training progressive layer dropping (PLD) gradual stacking MSG LEMON bert2BERT
Builds on: progressive layer dropping gradual stacking stochastic depth MSG LEMON
Closest papers by meaning
- Evolving Subnetwork Training for Large Language Models (ICML 2024 · similarity 0.88)
- PolyPythias: Stability and Outliers across Fifty Language Model Pre-Training Runs (ICLR 2025 · similarity 0.88)
- The Journey Matters: Average Parameter Count over Pre-training Unifies Sparse and Dense Scaling Laws (ICLR 2025 · similarity 0.88)
- Layer as Puzzle Pieces: Compressing Large Language Models through Layer Concatenation (NeurIPS 2025 · similarity 0.88)
- Model Merging in Pre-training of Large Language Models (NeurIPS 2025 · similarity 0.87)
- Through the River: Understanding the Benefit of Schedule-Free Methods for Language Model Training (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.