Research map / Optimization / Optimizers
Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add weight normalization to the network | ConvPool-CNN-C on CIFAR-10 · CIFAR-10 (no augmentation) | test error | No clear effect |
| add weight normalization combined with mean-only batch normalization (ablation) | ConvPool-CNN-C on CIFAR-10 · CIFAR-10 (no augmentation) | test error: best 7.31% vs ≈8.5% baseline | Helped |
| add weight normalization to mean-only batch normalization versus mean-only BN with normal parameterization (ablation) | ConvPool-CNN-C on CIFAR-10 · CIFAR-10 (no augmentation) | test error: 7.31% vs ≈8.5% | Helped |
| add weight normalization to DRAW LSTM weight vectors | DRAW on MNIST · MNIST test | marginal log likelihood lower bound convergence speed | Helped |
| add weight normalization to DQN | DQN on Atari (Space Invaders and others) · Atari games | evaluation score | Helped |
| add weight normalization to convolutional VAE | CVAE on MNIST and CIFAR-10 · MNIST and CIFAR-10 test lower bound | lower bound and variance across seeds | Helped |
| add batch normalization to CIFAR-10 ConvPool-CNN | ConvPool-CNN-C on CIFAR-10 · CIFAR-10 (no augmentation) | test error: 8.05% vs ≈8.5% | Helped |
| use exponential parameterization g = e^s instead of direct g (ablation) | weight normalization · CIFAR-10 (no augmentation) | test error and convergence speed | No clear effect |
| add data-dependent initialization | weight normalization · CIFAR-10 and other models | optimization robustness | Helped |
Closest papers by meaning
- Norm matters: efficient and accurate normalization schemes in deep networks (NeurIPS 2018 · similarity 0.91)
- Demystifying Batch Normalization in ReLU Networks: Equivalent Convex Optimization Models and Implicit Regularization (ICLR 2022 · similarity 0.91)
- Optimization Theory for ReLU Neural Networks Trained with Normalization Layers (ICML 2020 · similarity 0.90)
- Theoretical Analysis of Auto Rate-Tuning by Batch Normalization (ICLR 2019 · similarity 0.89)
- Implicit Regularization and Convergence for Weight Normalization (NeurIPS 2020 · similarity 0.89)
- RECAST: Reparameterized, Compact weight Adaptation for Sequential Tasks (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.