Research map / Large language models / Language, safety and interpretability
Deep Fusion: Efficient Network Training via Pre-trained Initializations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Deep Fusion, which combines pre-trained smaller T5 models into a larger wider network via a block-diagonal fusion operator, and uses backward error analysis with a Lie bracket term to explain the dynamics of mid-training network growth.
Problem
Training large transformers (LLMs) is computationally expensive, and network growing methods that enlarge models mid-training are poorly understood and inconsistent in performance.
Approach
Fuse two or more pre-trained smaller networks (or a model with itself, self fusion) into one wider network by block-diagonal weight composition with zero-initialized extra parameters, so the fused model initially equals an ensemble average; backward error analysis then shows the post-fusion dynamics carry a Lie bracket bias between small- and big-model gradients.
Key results
- Self fusion of T5-Small into T5-Medium reduced compute by 18% (58.4h vs 71.2h TPU hours) to reach baseline accuracy 67.2.
- Self fusion of T5-Small trained 500K steps then fine-tuned on GLUE achieved 84.94 GLUE average vs 84.74 for baseline trained 1M steps, at 29.9h vs 39.2h TPU time.
- Fusing T5-L (setting 5) gave a 20% reduction in computation time (197.9h vs 246.7h) with equal accuracy 70.57.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Maintain the fusion property rather than the fusion rule (ablation) | T5-Medium fusion · C4 validation | accuracy: 66.88 -> 67.25 ± 0.03 | Helped |
| Use self-fusion instead of fusing two distinct small models (ablation) | T5-Medium fusion · C4 validation | accuracy: 67.25 ± 0.03 -> 67.20 ± 0.05 | No clear effect |
| Use self-fusion instead of random initialization | T5-Medium · C4 | compute time: 71.2h -> 58.4h | Helped |
| Use self-fusion instead of random initialization | T5-Medium · C4 | accuracy: 67.2 -> 67.2 | No clear effect |
| Use self-fusion before fine-tuning rather than train T5-Small to 1M steps | T5-Small downstream model · GLUE average | GLUE average: 80.28 -> 81.01 | Helped |
| Use one self-fusion instead of training T5-L without fusion | T5-L · C4 | compute time: 246.7h -> 197.9h | Helped |
| Use one self-fusion instead of training T5-L without fusion | T5-L · C4 | accuracy: 70.57 -> 70.57 | No clear effect |
| Apply repeated self-fusion rather than one self-fusion (ablation) | T5-L · C4 | accuracy: 70.57 -> 70.47 | Hurt |
| Apply multiple regular fusions rather than repeated self-fusion (ablation) | T5-L · C4 | accuracy: 70.47 -> 70.74 | Helped |
| Use self-fusion rather than regular fusion (ablation) | T5-Medium downstream model · GLUE average | GLUE average: 84.86±0.13 -> 84.94$\pm$0.2 | Helped |
| Use self-fusion rather than random initialization | T5-Medium downstream model · GLUE average | compute time: 39.2h -> 29.9h | Helped |
| Use self-fusion rather than random initialization | T5-Medium downstream model · GLUE average | GLUE average: 84.74 -> 84.94 | Helped |
Surprising findings
- Repeated self fusion reduced performance, while repeated regular fusion improved T5-L performance.
- Higher learning rates right after fusion hurt short-term performance but benefited long-run results, contrary to preserving training dynamics after growth.
Limitations
- Experiments fuse only models trained on the same data with identical architectures.
- Fusion requires extra code changes for layer normalization; the fusion rule and fusion property cannot both be satisfied for normalization.
- Results are reported on T5 models only, with a single architecture family and limited scale.
Experimental setup
Datasets: C4 GLUE
Benchmarks: GLUE
Metrics: accuracy GLUE average validation loss wall-clock TPU hours
Baselines: baseline training from random initialization staged training
Builds on: T5 backward error analysis network morphism staged training
Closest papers by meaning
- The Journey Matters: Average Parameter Count over Pre-training Unifies Sparse and Dense Scaling Laws (ICLR 2025 · similarity 0.89)
- The Curse of Depth in Large Language Models (NeurIPS 2025 · similarity 0.88)
- FFN Fusion: Rethinking Sequential Computation in Large Language Models (NeurIPS 2025 · similarity 0.88)
- Dynamic Sparse No Training: Training-Free Fine-tuning for Sparse LLMs (ICLR 2024 · similarity 0.88)
- Benchmarking Optimizers for Large Language Model Pretraining (NeurIPS 2026 · similarity 0.87)
- SkipGPT: Each Token is One of a Kind (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.