Research map / Architectures and efficiency / RNNs and normalization
On the Importance of Gaussianizing Representations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes normality normalization, a normalization layer that applies a Newton-Raphson-estimated power transform to gaussianize pre-activations and adds scaled Gaussian noise during training, improving generalization across models and datasets.
Problem
Normalization layers control mean and variance of activations but do not prescribe what distribution activations should take; the paper asks whether encouraging Gaussian (normal) activations improves generalization and noise robustness.
Approach
Motivated by information-theoretic properties of the normal distribution (mutual information game, maximum entropy, uncorrelatedness implying independence), it augments existing normalization layers with a power transform whose parameter is estimated by one Newton-Raphson step on a quadratic NLL approximation, followed by additive Gaussian noise scaled by the channel's l1 dispersion, adding no lea
Key results
- ViT with LayerNormalNorm improves ImageNet Top1 from 71.54 (LN) to 75.25 and CIFAR100 from 66.40 to 70.12.
- BatchNormalNorm improves ResNet18 on CIFAR100 from 62.02 (BN) to 65.82 validation accuracy.
- BatchNormalNorm improves STL10 ResNet34 from 58.82 to 63.86 and gives lower relative noise-propagation error across layers than BatchNorm.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace LayerNorm with LayerNormalNorm | ViT · SVHN, CIFAR10, CIFAR100, Food101, ImageNet | validation accuracy: 94.61 ± 0.31 -> 95.78 ± 0.21; 89.97 ± 0.16 -> 91.18 ± 0.13; 66.40 ± 0.42 -> 70.12 ± 0.22; 73.25 ± 0.19 -> 79.11 ± 0.09; | Helped |
| Replace BatchNorm with BatchNormalNorm | ResNet18/34/50 · CIFAR10, CIFAR100, STL10, TinyIN, Caltech101, Food101 | validation accuracy: 88.89 ± 0.07 -> 90.41 ± 0.09; 62.02 ± 0.17 -> 65.82 ± 0.11; 58.82 ± 0.52 -> 63.86 ± 0.45; 58.22 ± 0.12 -> 60.57 ± 0.14; | Helped |
| Replace decorrelated BatchNorm with decorrelated BatchNormalNorm | ResNet18/34 · CIFAR10, CIFAR100, STL10 | validation accuracy: 90.66 ± 0.05 -> 91.50 ± 0.03; 65.11 ± 0.06 -> 67.53 ± 0.10; 66.76 ± 0.29 -> 69.36 ± 0.14 | Helped |
| Use scaled additive Gaussian noise rather than Gaussian dropout (ablation) | BatchNormalNorm · STL10 | validation accuracy | Helped |
| Scale additive Gaussian noise using channel minibatch statistics rather than fixed scaling (ablation) | BatchNormalNorm · STL10 | validation accuracy | Helped |
| Increase gaussianization strength toward the Newton-Raphson solution (alpha=1) (ablation) | BatchNormalNorm without noise, ResNet50 · Caltech101 | validation accuracy | Helped |
| Add the power transform (ablation) | BatchNorm baseline · Reported model and dataset combinations | validation accuracy | Helped |
| Add scaled additive Gaussian noise (ablation) | BatchNormalNorm without noise · Reported model and dataset combinations | validation accuracy | Helped |
| Train with BatchNormalNorm rather than BatchNorm | ResNet18/34 · Test-time random-noise robustness | relative error: 0.044 ± 0.002 vs 0.160 ± 0.009 | Helped |
Surprising findings
- Gaussian dropout and additive noise without scaling were less effective than the proposed scaled additive noise, suggesting the channel norm matters.
Limitations
- Increases runtime relative to standard normalization layers.
- Gaussianization is only a distributional defense and does not address adversarial robustness with exact weight knowledge.
Experimental setup
Datasets: CIFAR10 CIFAR100 STL10 TinyImageNet Caltech101 Food101 SVHN ImageNet
Benchmarks: CIFAR10 CIFAR100 STL10 TinyImageNet Caltech101 Food101 SVHN ImageNet
Metrics: validation accuracy relative error (noise robustness) Q-Q plot R^2 (gaussianity) adjusted mutual information HZ-statistic
Baselines: BatchNorm LayerNorm InstanceNorm GroupNorm decorrelated BatchNorm Gaussian dropout
Builds on: batch normalization layer normalization gaussian dropout power transform (Yeo-Johnson)
Closest papers by meaning
- Normalization Propagation: A Parametric Technique for Removing Internal Covariate Shift in Deep Networks (ICML 2016 · similarity 0.87)
- The Normalization Method for Alleviating Pathological Sharpness in Wide Neural Networks (NeurIPS 2019 · similarity 0.87)
- Normalizing the Normalizers: Comparing and Extending Network Normalization Schemes (ICLR 2017 · similarity 0.87)
- On the impact of activation and normalization in obtaining isometric embeddings at initialization (NeurIPS 2023 · similarity 0.86)
- Towards Training Without Depth Limits: Batch Normalization Without Gradient Explosion (ICLR 2024 · similarity 0.85)
- On the Nonlinearity of Layer Normalization (ICML 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.