Research map / Architectures and efficiency / RNNs and normalization
How to Train Your Multi-Exit Model? Analyzing the Impact of Training Strategies
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Analyzes training regimes for multi-exit (early-exit) networks and proposes a mixed regime (train backbone first, then jointly train backbone and internal classifiers), showing it beats joint and disjoint training across architectures, datasets, and exit methods.
Problem
Early-exit models are trained either jointly (backbone and internal classifiers together) or disjointly (backbone frozen, classifiers trained after), but the impact of this choice on performance across computational budgets is largely unexplored.
Approach
Proposes a mixed training regime that first trains the backbone to convergence and then trains the whole multi-exit network jointly, and introduces analysis tools (gradient dominance, mode connectivity, numerical rank, mutual information) to explain regime differences.
Key results
- Mixed regime achieves the best accuracy at medium and high budgets, e.g., ResNet-34 on CIFAR-100 reaches 75.88% at 100% budget vs 74.17% for joint and 73.79% for disjoint.
- On ViT-T ImageNet-1k at 50% budget, mixed reaches 62.96% vs 61.89% joint and 35.15% disjoint.
- Mixed regime converged in 1166 epochs on TinyImageNet ResNet-50 vs 1610 for joint training, while disjoint took 523 epochs.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace joint training with mixed training | ResNet-34 multi-exit model · CIFAR-100, 75% budget | accuracy: 74.32 -> 75.92 | Helped |
| Replace joint training with mixed training | ViT-T multi-exit model · CIFAR-100, 75% budget | accuracy: 66.22 -> 70.09 | Helped |
| Replace joint training with mixed training | ViT-T multi-exit model · ImageNet-1k, 75% budget | accuracy: 68.08 -> 70.79 | Helped |
| Replace joint training with mixed training | BERT-B multi-exit model · STS-B, 50% budget | MSE: 0.61 -> 0.59 | Helped |
| Replace joint training with mixed training | ResNet-50 multi-exit model · TinyImageNet, 25% budget | accuracy: 52.98 -> 52.89 | No clear effect |
| Add gradient equilibrium scaling to joint training (ablation) | ResNet-50 multi-exit model · TinyImageNet, higher computational budgets | accuracy: +0.73 | Helped |
| Add gradient equilibrium scaling to mixed training (ablation) | ResNet-50 multi-exit model · TinyImageNet | accuracy: -1.16 | Hurt |
| Increase loss weights along depth in mixed training (ablation) | ResNet-50 multi-exit model · TinyImageNet | accuracy: -1.08; -0.27; +0.22; +0.25 | Mixed |
| Decrease loss weights along depth in mixed training (ablation) | ResNet-50 multi-exit model · TinyImageNet | accuracy: +0.69; +0.32; -0.44; -0.46 | Mixed |
| Use larger internal classifiers in mixed training (ablation) | ViT-T multi-exit model · TinyImageNet, lower budgets | top-1 accuracy | Hurt |
| Use a two-layer 1024-unit head instead of a single-layer head (ablation) | ViT-T multi-exit model with mixed training · Imagenette, 25% budget | accuracy: 82.21 -> 80.25 | Hurt |
| Replace joint training with mixed training | ViT-T multi-exit model · CIFAR-100, PBEE, 75% budget | accuracy: 64.31 -> 68.33 | Helped |
Surprising findings
- Joint training shows a performance gap at higher computational budgets relative to mixed training, despite being the most common approach.
- Gradient equilibrium improves joint regime but gives no benefit to the mixed regime.
Limitations
- Disjoint regime is preferred only for the highest budgets in some setups and is weaker at low budgets.
- Mixed regime is not uniformly best; at low budgets joint training can match or exceed it (e.g., ResNet-34 CIFAR-100 at 25%).
Experimental setup
Datasets: CIFAR-100 ImageNet-1k TinyImageNet Imagenette 20-Newsgroups STS-B
Benchmarks: CIFAR-100 ImageNet-1k TinyImageNet 20-Newsgroups STS-B Imagenette
Metrics: top-1 accuracy MSE accuracy under computational budget (FLOPs %) numerical rank mutual information training epochs
Baselines: Joint training Disjoint training Branch-wise training Separate training Alternating training Loss scaling (SDN, Inc., Dec.) Gradient equilibrium
Builds on: Shallow-Deep Networks MSDNet gradient equilibrium PBEE GPF
Closest papers by meaning
- Jointly-Learned Exit and Inference for a Dynamic Neural Network (ICLR 2024 · similarity 0.87)
- Fast yet Safe: Early-Exiting with Risk Control (NeurIPS 2024 · similarity 0.86)
- Efficient stagewise pretraining via progressive subnetworks (ICLR 2025 · similarity 0.85)
- Beyond Greedy Exits: Improved Early Exit Decisions for Risk Control and Reliability (NeurIPS 2025 · similarity 0.84)
- BEEM: Boosting Performance of Early Exit DNNs using Multi-Exit Classifiers as Experts (ICLR 2025 · similarity 0.84)
- Instance-dependent Early Stopping (ICLR 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.