Research map / Architectures and efficiency / RNNs and normalization
Improving Training of Deep Neural Networks via Singular Value Bounding
Check your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Bound weight singular values with SVB | ConvNet · CIFAR10, 20-layer | error rate: 9.21 -> 8.03 | Helped |
| Bound weight singular values with SVB | ConvNet · CIFAR10, 38-layer | error rate: 12.08 -> 9.90 | Helped |
| Replace BN with BBN | SVB-trained ConvNet · CIFAR10, 20-layer | error rate: 8.03 -> 7.85 | Helped |
| Replace BN with BBN | SVB-trained ConvNet · CIFAR10, 38-layer | error rate: 9.90 -> 9.66 | Helped |
| Add SVB to SGD with momentum and BN (ablation) | 68-layer pre-activation ResNet · CIFAR10 | error rate (%): 6.10 -> 5.65 | Helped |
| Replace BN with BBN (ablation) | SVB-trained 68-layer pre-activation ResNet · CIFAR10 | error rate (%): 5.65 -> 5.37 | Helped |
| Use SVB and BBN (ablation) | Inception-ResNet · CIFAR10 | error rate (%): 5.40 -> 4.17 | Helped |
| Use SVB and BBN (ablation) | Inception-ResNet · CIFAR100 | error rate (%): 21.48 -> 18.30 | Helped |
| Use SVB and BBN (ablation) | Wide ResNet · CIFAR10 | error rate (%): 4.50 -> 3.58 | Helped |
| Use SVB and BBN (ablation) | Wide ResNet · CIFAR100 | error rate (%): 20.78 -> 18.32 | Helped |
| Use SVB and BBN (ablation) | Wider ResNet · CIFAR10 | error rate (%): 3.64 -> 3.06 | Helped |
| Use SVB and BBN (ablation) | Wider ResNet · CIFAR100 | error rate (%): 19.25 -> 16.90 | Helped |
Closest papers by meaning
- Batch Normalization Orthogonalizes Representations in Deep Random Networks (NeurIPS 2021 · similarity 0.89)
- Demystifying Batch Normalization in ReLU Networks: Equivalent Convex Optimization Models and Implicit Regularization (ICLR 2022 · similarity 0.88)
- Norm matters: efficient and accurate normalization schemes in deep networks (NeurIPS 2018 · similarity 0.87)
- Convolutional Normalization: Improving Deep Convolutional Network Robustness and Training (NeurIPS 2021 · similarity 0.87)
- Norm-Based Generalisation Bounds for Deep Multi-Class Convolutional Neural Networks (AAAI 2021 · similarity 0.87)
- Towards Training Without Depth Limits: Batch Normalization Without Gradient Explosion (ICLR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.