Research map / Architectures and efficiency / RNNs and normalization
Skip Connections Eliminate Singularities
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add skip connections from each layer to all higher layers | Fully connected networks · CIFAR-100 (coarse labels), training | training accuracy | Helped |
| Add identity skip connections between adjacent layers | Fully connected networks · CIFAR-100, training | training accuracy | Helped |
| Add randomized target-bias regularization | 30-layer fully connected networks · CIFAR-10 and CIFAR-100, training | training accuracy | Helped |
| Replace identity skip connections with random dense orthogonal skip matrices (ablation) | Residual networks · CIFAR-10 and CIFAR-100, training | training accuracy | Helped |
| Subtract the identity matrix from the initial residual weights (ablation) | Residual network · CIFAR-100, training | training accuracy | Hurt |
| Reduce the orthogonality of the skip connectivity matrix (ablation) | Residual networks · CIFAR-10 and CIFAR-100, training | training accuracy | Hurt |
| Increase correlations among skip vectors while keeping the skip-matrix eigenvalue spectrum fixed (ablation) | Residual networks · CIFAR-10 and CIFAR-100, training | training accuracy | Hurt |
| Replace randomized target-bias regularization with standard l2 bias regularization (ablation) | 30-layer fully connected networks · CIFAR-10 and CIFAR-100, training | training accuracy | No clear effect |
| Add a batch normalization layer at layer 15 (ablation) | BiasReg network · CIFAR-100, training | training accuracy | Helped |
Closest papers by meaning
- How Does Topology Influence Gradient Propagation and Model Performance of Deep Networks With DenseNet-Type Skip Connections? (CVPR 2021 · similarity 0.89)
- On skip connections and normalisation layers in deep optimisation (NeurIPS 2023 · similarity 0.88)
- Skip Connections Matter: On the Transferability of Adversarial Examples Generated with ResNets (ICLR 2020 · similarity 0.88)
- The Shattered Gradients Problem: If resnets are the answer, then what is the question? (ICML 2017 · similarity 0.87)
- Beyond Skip Connection: Pooling and Unpooling Design for Elimination Singularities (AAAI 2025 · similarity 0.87)
- Lambda-Skip Connections: the architectural component that prevents Rank Collapse (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.