Research map / Architectures and efficiency / RNNs and normalization
Controllable Orthogonalization in Training DNNs
Check your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Maintain layer-wise orthogonality with ONI | VGG-style networks · CIFAR-10, nine depth/width configurations | test error (%): ONI is lower than OLM-√2 in 8 of 9 configurations | Helped |
| Maintain layer-wise orthogonality with ONI | ResNet-110 without BN · CIFAR-10, batch size 128 | test error (%): 6.82 -> 6.56 | Helped |
| Maintain layer-wise orthogonality with ONI | ResNet-110 without BN · CIFAR-10, batch size 2 | test error (%): 7.24 -> 6.67 | Helped |
| Maintain layer-wise orthogonality with ONI | VGG-16 · ImageNet validation | Top-1 error (%): 27.47 -> 26.31 | Helped |
| Apply ONI to a ResNet without BN | 18-layer ResNet · ImageNet | top-1 test error (%): 33.84 -> 31.17 | Helped |
| Apply ONI to ResNet-50 | ResNet · ImageNet | test error (%): 23.85 -> 23.55 | Helped |
| Apply ONI to ResNetVar | ResNetVar · ImageNet, ResNet-101 | test error (%): 22.76 -> 21.89 | Helped |
| Replace spectral normalization with ONI (T=2) | DCGAN · CIFAR-10 | FID: 23.31 -> 20.75 | Helped |
| Scale orthogonal weights by √2 (ablation) | 20-layer MLP · Fashion-MNIST | test error | Helped |
| Scale orthogonal weights by √2 (ablation) | 6-layer MLP · Fashion-MNIST | test error | No clear effect |
| Use a small or large iteration count instead of an intermediate T (ablation) | ONI · Fashion-MNIST MLP, CIFAR-10 VGG-style, and ImageNet ResNet | test error | Hurt |
| Increase iteration count beyond T=2 (ablation) | ONI · CIFAR-10 DCGAN | FID | Hurt |
Closest papers by meaning
- On orthogonality and learning recurrent networks with long term dependencies (ICML 2017 · similarity 0.87)
- Orthogonal Weight Normalization: Solution to Optimization Over Multiple Dependent Stiefel Manifolds in Deep Neural Networks (AAAI 2018 · similarity 0.87)
- An Adaptive Orthogonal Convolution Scheme for Efficient and Flexible CNN Architectures (ICML 2025 · similarity 0.84)
- Preventing Dimensional Collapse in Self-Supervised Learning via Orthogonality Regularization (NeurIPS 2024 · similarity 0.83)
- Understanding Gradient Orthogonalization for Deep Learning via Non-Euclidean Trust-Region Optimization (NeurIPS 2026 · similarity 0.83)
- Convergence Analysis of Newton's Method for Neural Networks in the Overparameterized Limit (NeurIPS 2026 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.