Research map / Label-efficient and robust learning / Noisy labels
Ghost Noise for Regularizing Deep Neural Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Disentangles the regularizing "ghost noise" of Ghost Batch Normalization from its train-test discrepancy and proposes Ghost Noise Injection (GNI), which injects BN-style noise from randomly sampled ghost batches and outperforms GBN on CIFAR and ImageNet.
Problem
Ghost Batch Normalization improves generalization via noise in normalization statistics, but small normalization batches also cause a train-test discrepancy from self-dependency, creating a trade-off that limits the regularization benefit.
Approach
Decompose GBN into a double normalization, then propose GNI which computes shift and scale noise from randomly sampled ghost batches of unnormalized inputs (not the sample's own ghost batch) and injects it without normalizing over small batches; also XBN, which excludes each sample from its own statistics.
Key results
- On CIFAR-100 ResNet-18, GNI-16 reaches 78.84% mean test accuracy vs 77.10% for BN and 78.20% for GBN-16.
- On ImageNet-1k ResNet-50, GNI-64 reaches 77.98% vs 77.43% for BN-256 and 77.48% for GBN-64.
- GNI gives gains across NF ResNet, Simple ViT and ConvMixer on CIFAR-10, e.g. Simple ViT without augmentation 71.45% to 74.36%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace BN with GNI-16 | ResNet-18 · CIFAR-100 test | accuracy (%): 77.10 -> 78.84 | Helped |
| Use GNI-16 instead of GBN-16 | ResNet-18 · CIFAR-100 test | accuracy (%): 78.20 -> 78.84 | Helped |
| Add GNI-16 | ResNet-20 · CIFAR-10 test | accuracy (%): 92.22$\pm$0.05 -> \textbf{93.01}$\pm$0.28 | Helped |
| Add GNI-64 | ResNet-50 · ImageNet-1k test | accuracy (%): 77.43 -> \textbf{77.98} | Helped |
| Add GNI | ResNet-18 with WSLN · CIFAR-100 validation | accuracy | Helped |
| Add GNI to an NF ResNet | NF ResNet · CIFAR-10 test without augmentation | accuracy (%): 84.31$\pm$0.86 -> 86.18$\pm$0.58 | Helped |
| Add GNI to a Simple ViT | Simple ViT · CIFAR-10 test without augmentation | accuracy (%): 71.45$\pm$0.12 -> 74.36$\pm$0.27 | Helped |
| Add GNI to ConvMixer | ConvMixer · CIFAR-10 test without augmentation | accuracy (%): 91.18$\pm$0.24 -> 91.57$\pm$0.08 | Helped |
| Remove scaling noise (use shift-only GNI) (ablation) | GNI on ResNet-18 · CIFAR-100 validation | accuracy | Hurt |
| Remove shift noise (use scaling-only GNI) (ablation) | GNI on ResNet-18 · CIFAR-100 validation | accuracy | Hurt |
| Replace sampled channel-specific GNI noise with analytical GNI (ablation) | GNI on ResNet-18 · CIFAR-100 validation | accuracy | Hurt |
| Replace sampled GNI with dropout variants (ablation) | ResNet-18 · CIFAR-100 validation | accuracy | Hurt |
Surprising findings
- XBN, which removes self-dependency, can be unstable at small batch sizes because its output is unbounded.
- Analytical GNI with sampled noise from derived distributions underperforms sample-based GNI, suggesting channel-specific noise fluctuations matter.
Limitations
- Computational overhead from sampling ghost batches, up to roughly 50% slower than native BN without optimized kernels.
- Experiments limited mainly to ResNet-18 on CIFAR-100 due to compute; many repeated runs needed.
- Batch dependency remains; mitigation via history of statistics is only suggested.
Experimental setup
Datasets: CIFAR-100 CIFAR-10 ImageNet-1k MNIST
Benchmarks: CIFAR-100 CIFAR-10 ImageNet-1k
Metrics: top-1 test accuracy
Baselines: BN GBN XBN EvalNorm Dropout variants AGNI EAGN NoMorelization
Builds on: Batch Normalization Ghost Batch Normalization EvalNorm Gaussian Dropout Weight Standardization
Closest papers by meaning
- Four Things Everyone Should Know to Improve Batch Normalization (ICLR 2020 · similarity 0.90)
- Regularizing Deep Neural Networks by Noise: Its Interpretation and Optimization (NeurIPS 2017 · similarity 0.89)
- Towards Understanding Regularization in Batch Normalization (ICLR 2019 · similarity 0.89)
- Instance Enhancement Batch Normalization: An Adaptive Regulator of Batch Noise (AAAI 2020 · similarity 0.89)
- Deconstructing the Regularization of BatchNorm (ICLR 2021 · similarity 0.89)
- Understanding Batch Normalization (NeurIPS 2018 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.