Research map / Architectures and efficiency / RNNs and normalization
Masked Structural Growth for 2x Faster Language Model Pre-training
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Masked Structural Growth (MSG), a progressive pre-training framework that grows Transformers along all four dimensions with strictly function-preserving masked operators, achieving 2.2x speedup on Bert-large and 1.4x on GPT-2.
Problem
Pre-training large language models is computationally expensive; progressive growth from small to large Transformers needs good growth schedules and growth operators, but existing operators are not strictly function-preserving and depend on new-weight initialization.
Approach
Use external masks that zero out new neurons, heads, or layers immediately after growth (ensuring strict function preservation, including under LayerNorm), then linearly ramp the masks to 1; grow all four Transformer dimensions one at a time via a grid-searched schedule, independent of new-weight initialization.
Key results
- MSG Sch1-L reaches GLUE avg 83.2 vs 82.2 for full training of Bert-large with 2.2x speedup (32h10min vs 70h48min).
- On Bert-base MSG matches full training (GLUE 81.0 vs 80.7) at 1.8x speedup (14h36min vs 26h10min).
- On GPT-2 MSG gets 1.4x speedup with zero-shot Wikitext2 PPL 41.20 vs 41.31 for full training.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| remove masks after growth (ablation) | MSG on Bert-Large with Sch1-L · GLUE Avg. | score: 83.2(0.2) -> 82.2(0.2) | Hurt |
| remove masks after growth (ablation) | MSG on Bert-Large with Rapid-L · GLUE Avg. | score: 79.1(0.4) -> 77.9(0.3) | Hurt |
| remove masks after growth (ablation) | MSG on Bert-Large with Sch1-L · SQuADv1.1 accuracy | accuracy: 81.3(0.2) -> 79.7(0.3) | Hurt |
| remove masks after growth (ablation) | MSG on Bert-Large with Rapid-L · SQuADv1.1 accuracy | accuracy: 76.5(0.6) -> 75.7(0.4) | Hurt |
| use rapid successive growth stages instead of Sch1-L stage durations (ablation) | MSG on Bert-Large · GLUE Avg. | score: 83.2(0.2) -> 79.1(0.4) | Hurt |
| use Sch0-L intermediate structures instead of Sch1-L (ablation) | MSG on Bert-Large · GLUE Avg. | score: 83.2 vs. 82.8 | Hurt |
| use Sch0-B intermediate structures instead of Sch1-B (ablation) | MSG on Bert-Base · GLUE Avg. | score: 80.9 vs. 81.0 | No clear effect |
| increase mask ramp duration φ beyond 500 steps (ablation) | MSG on Bert-Large · compressed Rapid-L | loss@10k: 3.23 -> 3.23 | No clear effect |
| use MSG operators instead of Bert2BERT operators | Bert-Large with Sch1-L · GLUE Avg. | score: 82.1(0.3) -> 83.2(0.2) | Helped |
| use MSG operators instead of Bert2BERT operators | GPT-2 with Sch1-G · Wikitext2 validation | zero-shot perplexity: 42.24 -> 41.20 | Helped |
| use MSG operators instead of Bert2BERT operators | GPT-2 with Sch1-G · Wikitext2 validation | fine-tuned perplexity: 25.13 -> 24.14 | Hurt |
| use MSG progressive growth instead of full training | GPT-2 · Wikitext2 validation | fine-tuned perplexity: 23.90 -> 24.14 | Hurt |
Surprising findings
- Removing the function-preserving mask (No Mask random init) performs comparably to full training on the moderate schedule, suggesting random initialization can beat weight-dependent Bert2BERT.
- Rapid multi-expansion schedules within a short span hurt Transformer training notably (Rapid-L GLUE 79.1 vs 83.2 for Sch1-L).
Limitations
- Schedules are heuristic and grow only one dimension per stage; combining dimensions in the same stage is unexplored.
- Best initialization strategy for Transformer growth remains open.
- Grid search for schedules is hand-guided and not automated or self-adaptive.
Experimental setup
Datasets: English Wikipedia BookCorpus OpenWebText GLUE SQuAD v1.1 WikiText-2
Benchmarks: GLUE SQuAD v1.1 WikiText-2
Metrics: GLUE average score accuracy F1 perplexity (zero-shot and fine-tuned) wall-clock speed-up
Baselines: Bert2BERT Net2Net No Mask ablation Full training from scratch LiGO CompoundGrow
Builds on: Net2Net Bert2BERT CompoundGrow stacking LiGO
Closest papers by meaning
- Stacking Your Transformers: A Closer Look at Model Growth for Efficient LLM Pre-Training (NeurIPS 2024 · similarity 0.90)
- Algorithmic progress in language models (NeurIPS 2024 · similarity 0.89)
- Instruction-Following Pruning for Large Language Models (ICML 2025 · similarity 0.89)
- Data-Constrained Language Model Pretraining: Improved Regularization and Scaling Laws (NeurIPS 2026 · similarity 0.88)
- The Journey Matters: Average Parameter Count over Pre-training Unifies Sparse and Dense Scaling Laws (ICLR 2025 · similarity 0.88)
- Pruning Large Language Models with Semi-Structural Adaptive Sparse Training (AAAI 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.