Research map / Architectures and efficiency / RNNs and normalization
Inherent Weight Normalization in Stochastic Neural Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace the StNN activation with multiplicative-noise threshold neurons and inherent weight normalization | fully connected MNIST network · PI MNIST test | classification error: 1.47 % -> 1.36 % | Helped |
| scale weights by .1 during inference (ablation) | NSM · PI MNIST test | classification error: 1.36 % -> 1.38 % | No clear effect |
| use Bernoulli multiplicative noise instead of a deterministic CNN | convolutional network · MNIST | classification error: 0.880% -> 0.775 % | Helped |
| use Gaussian multiplicative noise instead of a deterministic CNN | convolutional network · EMNIST | classification error: 6.938% -> 6.256% | Helped |
| use Bernoulli multiplicative noise instead of a deterministic CNN | convolutional network · N-MNIST | classification error: 0.927% -> 0.689 % | Helped |
| replace deterministic CNN with Bernoulli NSM | convolutional network · CIFAR10 | classification error: $10.47 \%$ -> $9.98 \%$ | Helped |
| replace deterministic CNN with Bernoulli NSM | convolutional network · CIFAR100 | classification error: $34.37 \%$ -> $34.85 \%$ | Hurt |
| use NSM with Bernoulli multiplicative noise | DVS Gestures convolutional network · DVS Gestures | classification error: 9.16% -> 8.56% | Helped |
| use straight-through gradients instead of activation-probability gradients (ablation) | bNSM · MNIST | classification error: $0.775$ -> $2.13$ | Hurt |
| add weight normalization to a deterministic binary network (ablation) | BD · MNIST | classification error: $3.11$ -> $2.72$ | Helped |
| train bNSM without pretrained weights (ablation) | bNSM · CIFAR100 | classification error: $34.85 \%$ -> $35.19 \%$ | Hurt |
| train gNSM without pretrained weights (ablation) | gNSM · CIFAR100 | classification error: $34.84 \%$ -> $34.93 \%$ | No clear effect |
Closest papers by meaning
- Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks (NeurIPS 2016 · similarity 0.89)
- Stochastic Normalization (NeurIPS 2020 · similarity 0.88)
- Norm matters: efficient and accurate normalization schemes in deep networks (NeurIPS 2018 · similarity 0.88)
- Understanding the Generalization of Stochastic Gradient Adam in Learning Neural Networks (NeurIPS 2025 · similarity 0.86)
- LEAPS: A discrete neural sampler via locally equivariant networks (ICML 2025 · similarity 0.86)
- Self-Normalized Resets for Plasticity in Continual Learning (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.