Research map / Architectures and efficiency / RNNs and normalization
A Multi-Level Framework for Accelerating Training Transformer Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a multi-level V-cycle training framework built from coalescing, de-coalescing and interpolation operators that trains a smaller coarsened transformer first and maps its parameters back to the larger model, cutting training FLOPs on BERT, GPT and DeiT.
Problem
Training large transformer models (BERT, GPT, ViT/DeiT) demands enormous compute and energy; existing growth-based methods only reuse small pre-trained models and ignore the cost of training them.
Approach
Borrowing the V-cycle from multigrid, the method progressively coalesces a model in width and depth, trains the small model for fast convergence, de-coalesces it back, and interpolates the de-coalesced parameters with the original model to break neuron symmetry before continuing training.
Key results
- Saves 19.0% FLOPs and 10.8% walltime for BERT-Base pre-training with GLUE average 79.8 vs 79.7 for training from scratch.
- Saves 24.1% FLOPs and 16.5% walltime for GPT-Base pre-training with comparable or better zero-shot perplexity.
- On BERT-Large, 3-level V-cycle saves 51.6% FLOPs and 41.9% walltime with GLUE average 81.5 vs 80.6 from scratch; DeiT-B saves 27.1% FLOPs and 24.3% walltime.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use multi-level V-cycle training instead of training from scratch | BERT-Base · BERT-Base GLUE | Avg: 79.7(0.2) -> 79.8(0.1) | No clear effect |
| Use multi-level V-cycle training instead of training from scratch | BERT-Base · BERT-Base pre-training | Walltime saving: 10.8% | Helped |
| Use multi-level V-cycle training instead of training from scratch | GPT-Base · GPT-Base LAMBADA zero-shot | perplexity: 54.5 -> 53.2 | Helped |
| Use multi-level V-cycle training instead of training from scratch | GPT-Base · GPT-Base pre-training | Walltime saving: 16.5% | Helped |
| Use multi-level V-cycle training instead of training from scratch | DeiT-B · DeiT-B ImageNet | Top 1 Acc: 81.1% -> 81.5% | No clear effect |
| Use a 3-level rather than 2-level V-cycle (ablation) | BERT-Large · BERT-Large pre-training | FLOPs saving: 37.4% -> 51.6% | Helped |
| Increase initialization training before coalescing to 50K steps (ablation) | BERT-Base multi-level training · BERT-Base pre-training | Walltime saving: 10.8% -> 1.0% | Hurt |
| Increase initialization training before coalescing to 100K steps (ablation) | BERT-Base multi-level training · BERT-Base pre-training | Walltime saving: 10.8% -> -7.4% | Hurt |
| Vary small-model training duration from 50K to 300K steps (ablation) | BERT-Base multi-level training · BERT-Base pre-training | FLOPs saving: 14.5%; 17.7%; 17.8%; 12.1% | No clear effect |
| Set interpolation ratio to 1.0, removing interpolation into the original model (ablation) | BERT-Base multi-level training · BERT-Base pre-training | FLOPs saving: -15.8% | Hurt |
| Set interpolation ratio to 0.05 (ablation) | BERT-Base multi-level training · BERT-Base pre-training | FLOPs saving: -2.3% | Hurt |
| Use interpolation ratio 0.25 instead of 0.5 (ablation) | BERT-Base multi-level training · BERT-Base pre-training | FLOPs saving: 19.0% -> 18.1% | No clear effect |
Surprising findings
- Repeatedly enlarging a model monotonically (mapped twice) converges significantly slower than mapping once, motivating the V-cycle design.
- Removing interpolation (alpha=1.0) yields no savings, and too small alpha gives negative savings.
Limitations
- Gains are smaller for DeiT-S (12.2% FLOPs) and for some settings walltime savings are negative, e.g., bert2BERT and KI in some settings.
- Three or more levels gave little gain on BERT-Base because the smallest model is too small to learn from the large dataset.
- Only V-cycle explored; W-cycle and FMG left for future work.
Experimental setup
Datasets: English Wikipedia BooksCorpus ImageNet GLUE LAMBADA PTB WikiText-2 WikiText103 CIFAR10 CIFAR100 Flowers102 Stanford Cars
Benchmarks: GLUE LAMBADA PTB WikiText-2 WikiText103 CIFAR10 CIFAR100 Flowers102 Stanford Cars ImageNet
Metrics: FLOPs saving walltime saving accuracy Matthews correlation perplexity
Baselines: training from scratch StackBERT bert2BERT LiGO Network Expansion KI
Builds on: multigrid method LiGO Net2Net StackBERT
Closest papers by meaning
- Transformers Get Stable: An End-to-End Signal Propagation Theory for Language Models (ICML 2024 · similarity 0.88)
- HybridNorm: Towards Stable and Efficient Transformer Training via Hybrid Normalization (NeurIPS 2025 · similarity 0.87)
- Preparing Lessons for Progressive Training on Language Models (AAAI 2024 · similarity 0.87)
- LEMON: Lossless model expansion (ICLR 2024 · similarity 0.87)
- GPAS: Accelerating Convergence of LLM Pretraining via Gradient-Preserving Activation Scaling (NeurIPS 2025 · similarity 0.87)
- Building on Efficient Foundations: Effective Training of LLMs with Structured Feedforward Layers (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.