Research map / Architectures and efficiency / RNNs and normalization
The Optimal Control Foundation of Early Exits – Turnpikes and ResNets
Jens Püttschneider, Simon Heilig, Asja Fischer, Timm Faulwasser
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Analyzes joint optimization of output-layer and intermediate-layer losses (early exits) in residual networks through an optimal control lens, linking it to turnpike properties.
Problem
Early-exit training jointly optimizes losses at the output and intermediate layers to trade off efficiency and accuracy, but the theoretical behavior of this joint optimization is not well understood.
Approach
Frames deep residual networks with intermediate losses as an optimal control problem and analyzes the joint optimization of intermediate and output losses using turnpike theory.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| increase stage-cost weight gamma to >=0.2 (ablation) | Early-exit ResNet-110 joint training · CIFAR-10 (N=54, M=1) | final-block performance | Hurt |
| use very small gamma (<=0.002) (ablation) | Early-exit ResNet-110 joint training · CIFAR-10 (N=54, M=1) | early-layer convergence | Hurt |
| share early-exit parameters with the backbone (ablation) | Early-exit ResNet · CIFAR-10 | test accuracy | Hurt |
| place exits every 6 blocks instead of every block (ablation) | Early-exit ResNet · CIFAR-10 | test accuracy | Improved |
| place exits every 6 blocks instead of every block (ablation) | Early-exit ResNet · CIFAR-10 | average exit depth (compute) | Hurt |
| use homogeneous (constant 64-filter) ResNet instead of standard (ablation) | Early-exit ResNet · CIFAR-10/CIFAR-100 | test accuracy | Hurt |
| add joint early-exit training (M=1) to ResNet-110 | ResNet-110 · TinyImageNet test | test accuracy (%): 62.4 -> 58.3 | Hurt |
| increase backbone depth (N=18,36,54) under joint training (ablation) | Early-exit ResNet · CIFAR-10 early-exit accuracy vs threshold | early-exit accuracy | No clear effect |
Closest papers by meaning
- Scaling ResNets in the Large-depth Regime (JMLR 2025 · similarity 0.88)
- Neural Collapse is Globally Optimal in Deep Regularized ResNets and Transformers (NeurIPS 2025 · similarity 0.87)
- Transformative or Conservative? Conservation laws for ResNets and Transformers (ICML 2025 · similarity 0.87)
- Loss Landscape of Shallow ReLU-like Neural Networks: Stationary Points, Saddle Escape, and Network Embedding (ICLR 2025 · similarity 0.85)
- Feature Learning Dynamics in Infinite-Depth Neural Networks (NeurIPS 2026 · similarity 0.85)
- Fast yet Safe: Early-Exiting with Risk Control (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
