Research map / Architectures and efficiency / RNNs and normalization
Evolving Subnetwork Training for Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Evolving Subnetwork Training (EST), which randomly samples subnetworks over layers, attention heads, and MLP width while progressively enlarging them to the full model, saving about 25-27% training FLOPs on GPT2 and TinyLlama.
Problem
Training large language models is very expensive because of their massive parameter counts, and existing incremental training methods save little cost when long training is needed.
Approach
Maintain the full model and randomly sample subnetworks (subsets of layers, attention heads, and MLP neurons, with 1/p rescaling) at each step, using a multi-stage sampling scheduler that gradually grows subnetwork size until the complete model is trained; the approach breaks loss-preserving and training-dynamics-preserving properties, which the authors argue is what yields savings.
Key results
- On GPT2-base, EST saves 26.7% training FLOPs and gives 1.22x wall-clock speedup, with validation loss 3.05 vs 3.06 for the original and average GLUE 80.66 vs 79.84.
- On TinyLlama 1.1B, EST saves 25% FLOPs with 1.22x speedup, loss 2.65 vs 2.64, and GPT4All average 42.79 vs 42.40.
- Ablations show the three-stage practical scheduler outperforms one-, two-stage, and three-stage variants at comparable cost; EST-one-stage (50% saving) loses accuracy (loss 3.36).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Progressively sample subnetworks, then train the complete model | GPT2-base · GLUE | average score: 79.84 -> 80.66 | Helped |
| Progressively sample subnetworks, then train the complete model | GPT2-base · LAMBADA | accuracy: 29.44 -> 32.01 | Helped |
| Progressively sample subnetworks, then train the complete model | TinyLlama 1.1B · GPT4All | average score: 42.40 -> 42.79 | Helped |
| Progressively sample subnetworks, then train the complete model | TinyLlama 1.1B · SlimPajama validation | loss: 2.64 -> 2.65 | No clear effect |
| Use one sampled stage and never train the complete model (ablation) | EST on GPT2-base · GLUE | average score: 80.66 -> 77.78 | Hurt |
| Skip stage 1 and transition at 50k (ablation) | EST on GPT2-base · GLUE | average score: 80.66 -> 81.05 | Helped |
| Skip stage 1 and transition at 70k (ablation) | EST on GPT2-base · GLUE | average score: 80.66 -> 80.17 | Hurt |
| Skip stage 1 and transition at 90k (ablation) | EST on GPT2-base · GLUE | average score: 80.66 -> 80.41 | Hurt |
| In stage 2, sample layers instead of MHA and MLP modules (ablation) | EST on GPT2-base · GLUE | average score: 80.66 -> 79.47 | Hurt |
| In stage 2, sample layers instead of MHA and MLP modules (ablation) | EST on GPT2-base · LAMBADA | accuracy: 32.01 -> 29.67 | Hurt |
Surprising findings
- A one-stage schedule that never trains the full model still saves 50% FLOPs but degrades loss to 3.36, showing a long final full-model stage is vital.
- Training with EST yields a much smaller Hessian trace (2510 vs 11763) at nearly the same loss, which the authors link to better generalization.
Limitations
- Sampling scheduler is hand-designed with a three-stage heuristic; no theory for choosing schedules.
- Experiments are limited to GPT2-base (117M) and TinyLlama (1.1B); scalability to larger models is untested.
- Speedups on GPU are below the theoretical FLOPs saving because of memory access overhead and matrix parallelism effects.
Experimental setup
Datasets: OpenWebText SlimPajama Starcoder GLUE SQuAD LAMBADA GPT4All
Benchmarks: GLUE SQuAD LAMBADA GPT4All
Metrics: validation loss accuracy F1 Matthews correlation Pearson/Spearman correlation wall clock time FLOPs saving Hessian trace
Baselines: Original naive training Staged Training MSG EST one-stage / two-stage variants
Builds on: Stochastic Depth Structural Dropout Net2net MSG Staged Training
Closest papers by meaning
- Efficient stagewise pretraining via progressive subnetworks (ICLR 2025 · similarity 0.88)
- Search for Efficient Large Language Models (NeurIPS 2024 · similarity 0.88)
- Model Merging in Pre-training of Large Language Models (NeurIPS 2025 · similarity 0.87)
- BLoB: Bayesian Low-Rank Adaptation by Backpropagation for Large Language Models (NeurIPS 2024 · similarity 0.87)
- Pruning Large Language Models with Semi-Structural Adaptive Sparse Training (AAAI 2025 · similarity 0.87)
- Deep Fusion: Efficient Network Training via Pre-trained Initializations (ICML 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.