Research map / Architectures and efficiency / RNNs and normalization
Adaptive Depth Networks with Skippable Sub-Paths
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes adaptive depth networks (ADN) that split each residual stage into a mandatory and a skippable sub-path, trained via a simple super-net-to-base-net self-distillation with skip-aware BNs/LNs, enabling combinatorial depth selection for CNNs and transformers.
Problem
Adaptive depth networks are hard and slow to train because prior methods self-distill exhaustively over many sub-networks, and there is no principled account of which layers can be skipped.
Approach
Divide each residual stage into a mandatory sub-path and a skippable sub-path; train only the super-net (teacher) and base-net (student) with KL self-distillation so the skippable blocks act as small feature refinements, then combine sub-paths at test time to get 2^N_r sub-networks.
Key results
- ResNet50-ADN (TTTT) reaches 76.1% Acc@1 at 2.6G FLOPs, versus 37% FLOPs reduction, matching DepGraph-style pruning baselines at higher accuracy than DR-ResNet50 (75.3%, 2.3G).
- Swin-T-ADN (TTTT) achieves 78.0% Acc@1 at 2.3G FLOPs (48% reduction).
- On Jetson Orin Nano, 38% FLOPs reduction via depth adaptation cuts latency and energy by 35%, while S-ResNet50 gets only up to 9% speedup despite 93% FLOPs reduction.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add self-distillation and skip-aware BNs (ablation) | ResNet50-ADN · ImageNet validation | Acc@1: 75.2% to 77.6%; 72.2% to 76.1% | Helped |
| Add self-distillation (ablation) | ResNet50-ADN with skip-aware BNs · ImageNet validation | Acc@1: 76.6% to 77.6%; 75.1% to 76.1% | Helped |
| Add skip-aware BNs (ablation) | ResNet50-ADN with self-distillation · ImageNet validation | Acc@1: 76.1% to 77.6%; 74.9 % to 76.1% | Helped |
| Add intermediate-feature distillation to output distillation (ablation) | ResNet50-ADN · ImageNet validation | Acc@1: 77.6% to 77.3%; 76.1% to 76.2% | No clear effect |
| Use FFFF as teacher and TTTT as student instead of random teacher/student sampling (ablation) | ResNet50-ADN · ImageNet validation | Acc@1: 77.1 to 77.6; 76.7 to 77.1; 76.4 to 76.5; 75.5 to 76.0; 74.8 to 76.1 | Helped |
| Use FFFF as teacher and TTTT as student instead of random teacher/student sampling (ablation) | ViT-b/32-ADN · ImageNet validation | Acc@1: 75.2 to 76.6; 74.7 to 76.0; 73.9 to 75.5; 72.9 to 74.5; 71.1 to 74.3 | Helped |
| Make mandatory sub-paths shorter than skippable sub-paths (ablation) | ResNet50-ADN · ImageNet validation | Acc@1 | Hurt |
| Increase mandatory sub-path lengths beyond the default ratio (ablation) | ResNet50-ADN · ImageNet validation | Acc@1 | No clear effect |
| Make mandatory sub-paths shorter by selecting the last two encoder blocks as skippable (ablation) | ViT-b/16-ADN · ImageNet validation | Acc@1 | Hurt |
| Use adaptive depth sub-networks instead of individually trained networks | ResNet50 · ImageNet validation | Acc@1 | Helped |
| Reduce FLOPs through depth adaptation | ResNet50-ADN on Jetson Orin Nano · On-device inference | inference latency: reducing FLOPs by 38% ... reduces ... inference latency ... by 35% | Helped |
| Use ResNet50-ADN as the detector backbone rather than individual ResNet50 | Faster R-CNN · MS COCO val2017 | Box AP: 36.4 to 37.8 | Helped |
Surprising findings
- Sub-networks formed by combining untrained sub-paths often outperform sub-networks explicitly trained with random self-distillation sampling.
- Shortening mandatory sub-paths in favor of skippable ones significantly degrades accuracy across all sub-networks.
Limitations
- Limited FLOPs adaptation range (only 2^N_r sub-networks, with the base-net roughly 37-48% cheaper than super-net)
- Mandatory sub-path length must be maintained; shorter mandatory paths degrade all sub-networks
- Stochastic depth must be disabled for Swin-T-ADN mandatory sub-paths due to conflicts
Experimental setup
Datasets: ImageNet ILSVRC2012 MS COCO 2017
Benchmarks: ImageNet ILSVRC2012 validation MS COCO val2017
Metrics: Top-1 accuracy (Acc@1) FLOPs Box AP Mask AP inference latency energy consumption
Baselines: ResNet50 (individual) ResNet50 (KD individual) AlphaNet S-ResNet50 GReg-1 DepGraph DR-ResNet50 AlphaNet-0.75x X-Pruner SPViT-Swin-Ti Ada-DeiT-S WDPruning
Builds on: ResNet Swin Transformer AlphaNet DeiT training recipe
Closest papers by meaning
- Adaptive Neural Networks for Efficient Inference (ICML 2017 · similarity 0.88)
- Hyperparameter Transfer Laws for Non-Recurrent Multi-Path Neural Networks (ICML 2026 · similarity 0.86)
- Depthwise Hyperparameter Transfer in Residual Networks: Dynamics and Scaling Limit (ICLR 2024 · similarity 0.85)
- Tensor Programs VI: Feature Learning in Infinite Depth Neural Networks (ICLR 2024 · similarity 0.85)
- Revisiting Residual Connections: Orthogonal Updates for Stable and Efficient Deep Networks (NeurIPS 2025 · similarity 0.85)
- Feature Learning Dynamics in Infinite-Depth Neural Networks (NeurIPS 2026 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.