Research map / Optimization / Implicit bias and generalization
Towards Understanding the Importance of Shortcut Connections in Residual Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add step size warmup: use a smaller inner-layer step size initially, then increase it (ablation) | ResNet · Synthetic teacher, k=16 and k=100, p=8 | success rate: 0.7042 -> 1.0000; 0.8426 -> 1.0000 | Helped |
| Remove step size warmup (ablation) | ResNet · Synthetic teacher, k=16 to k=100, p=8 | success rate: 1.0000 -> 0.7042; 1.0000 -> 0.8426 | Hurt |
| Use a CNN student without shortcut connections instead of the ResNet student | Two-layer student trained on the synthetic teacher · Synthetic teacher, k=16 to k=100, p=8 | success rate: 1.0000 -> 0.5348; 1.0000 -> 0.5374 | Hurt |
Closest papers by meaning
- Residual Alignment: Uncovering the Mechanisms of Residual Networks (NeurIPS 2023 · similarity 0.91)
- Are ResNets Provably Better than Linear Predictors? (NeurIPS 2018 · similarity 0.91)
- Residual Distillation: Towards Portable Deep Neural Networks without Shortcuts (NeurIPS 2020 · similarity 0.90)
- Residual Connections Encourage Iterative Inference (ICLR 2018 · similarity 0.90)
- Are deep ResNets provably better than linear predictors? (NeurIPS 2019 · similarity 0.90)
- Neural Collapse is Globally Optimal in Deep Regularized ResNets and Transformers (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.