Research map / Architectures and efficiency / RNNs and normalization
Revisiting Residual Connections: Orthogonal Updates for Stable and Efficient Deep Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Orthogonal Residual Update, which adds only the component of a module's output orthogonal to the residual stream, and reports gains on CIFAR, TinyImageNet and ImageNet-1k for ResNetV2 and ViT models.
Problem
Standard additive residual updates in networks with linear residual streams may mostly rescale or reinforce the existing stream direction rather than contributing novel features, underusing module capacity.
Approach
Decompose the module output relative to the input stream x_n and add only the orthogonal component f_perp = f(σ(x_n)) − s_n x_n, with s_n the scalar projection coefficient, yielding x_{n+1} = x_n + f_perp(x_n); the identity path is kept, and the module itself is not constrained to be orthogonal.
Key results
- ViT-B on ImageNet-1k: Orthogonal-F 77.05% vs Linear 73.27% Acc@1 (+3.78 pp).
- ViT-S on CIFAR-100: Orthogonal-F 73.86% vs Linear 71.92% Acc@1 over 5 runs.
- ResNetV2 gains are smaller; e.g., ResNetV2-50 on CIFAR-10 and CIFAR-100 are slightly lower or equal for Orthogonal-F (94.71 vs 94.75 and 77.43 vs 77.90).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace linear residual updates with feature-wise orthogonal updates | ViT-B · ImageNet-1k | Acc@1: 73.27{ $\pm$0.58} -> 77.05{ $\pm$0.21}; +3.78 pp | Helped |
| Replace linear residual updates with feature-wise orthogonal updates | ViT-B · CIFAR-10, CIFAR-100, TinyImageNet | Acc@1: 87.28{ $\pm$0.41} -> 91.93{ $\pm$0.08}; 68.25{ $\pm$0.88} -> 75.07{ $\pm$0.43}; 55.29{ $\pm$0.71} -> 57.87{ $\pm$0.37} | Helped |
| Replace linear residual updates with feature-wise orthogonal updates | ResNetV2-18/34 · CIFAR-10, CIFAR-100, TinyImageNet | Acc@1: ResNetV2-18: 95.06{ $\pm$0.15} -> \textbf{95.26}{ $\pm$0.12}; 77.67{ $\pm$0.28} -> \textbf{77.87}{ $\pm$0.27}; 62.04{ $\ | Helped |
| Replace linear residual updates with orthogonal updates | ResNetV2-50/101 · CIFAR-10, CIFAR-100, TinyImageNet | Acc@1: ResNetV2-50 CIFAR-10: \textbf{94.75}{ $\pm$0.09} -> \textbf{94.75}{ $\pm$0.10}; ResNetV2-50 CIFAR-100: \textbf{77.90}{ $ | Mixed |
| Increase the probability of using orthogonal connections (ablation) | ViT-S · TinyImageNet | Acc@1 and Acc@5 | Helped |
| Apply orthogonal updates in all 6 layers rather than none (ablation) | ViT-S · TinyImageNet | Acc@1: 51.88{ $\pm$0.42} -> \textbf{52.98}{ $\pm$0.45} | Helped |
| Switch from linear to orthogonal updates after linear training (ablation) | ViT-S · CIFAR-10, CIFAR-100, TinyImageNet | Acc@1: 92.78 -> 92.88; 74.22 -> 74.02; 53.24 -> 52.14 | No clear effect |
| Train orthogonally first, then switch to linear updates (ablation) | ViT-S · CIFAR-10, CIFAR-100, TinyImageNet | Acc@1: 92.78 -> 93.89; 74.22 -> 75.63; 53.24 -> 54.58 | Helped |
| Set the stability constant to epsilon=10^-6 (ablation) | Orthogonal Residual Update · ViT-S on TinyImageNet | run-to-run stability | Helped |
| Add a final LayerNorm before the classifier (ablation) | ResNetV2-34 · TinyImageNet | Acc@1: 65.88%{$\pm$0.22%} (Linear); 65.83%{$\pm$0.38%} (Orthogonal-F) | Mixed |
| Change initial learning rate across the reported sweep while keeping other hyperparameters identical (ablation) | ViT-S with orthogonal updates · CIFAR-10 and CIFAR-100 | Acc@1: 90.41 -> 90.56; 90.45 -> 90.91; 89.95 -> 90.36; 84.85 -> 87.38; 66.09 -> 72.20 | Helped |
Surprising findings
- ResNetV2-50 on CIFAR-10/CIFAR-100 shows no gain, and sometimes a small drop, from orthogonal updates.
- Adding a final LayerNorm before the classifier removes the ResNetV2-34 TinyImageNet advantage (65.88 linear vs 65.83 orthogonal-F), suggesting part of the gain was due to norm explosion in the linear baseline.
Limitations
- Compute-limited: models up to ViT-B and datasets up to ImageNet-1k; no LLMs or larger-scale tests.
- Hyperparameter sweeps are non-exhaustive, and the theoretical account of when orthogonal updates help is open.
- Module output can still accumulate a parallel component, and the epsilon term leaves a small parallel bias.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 TinyImageNet ImageNet-1k
Benchmarks: CIFAR-10 CIFAR-100 TinyImageNet ImageNet-1k
Metrics: Acc@1 Acc@5 effective rank spectral entropy linear CKA feature std dev training throughput
Baselines: Linear residual update (standard ResNetV2 / ViT) Learned stream-scaling variant
Builds on: ResNetV2 Vision Transformer AdamW Gram-Schmidt orthogonalization
Closest papers by meaning
- Residual Alignment: Uncovering the Mechanisms of Residual Networks (NeurIPS 2023 · similarity 0.90)
- Deep Residual Learning for Image Recognition (CVPR 2016 · similarity 0.87)
- LAuReL: Learned Augmented Residual Layer (ICML 2025 · similarity 0.87)
- Hyper-Connections (ICLR 2025 · similarity 0.86)
- Depthwise Hyperparameter Transfer in Residual Networks: Dynamics and Scaling Limit (ICLR 2024 · similarity 0.86)
- An Adaptive Orthogonal Convolution Scheme for Efficient and Flexible CNN Architectures (ICML 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.