Research map / Architectures and efficiency / RNNs and normalization
Fixup Initialization: Residual Learning Without Normalization
Hongyi Zhang, Yann N. Dauphin, Tengyu Ma
This paper tried a common change: Add mixup or CutMix
Add Mixup regularization to Fixup · ImageNet · test error (%): 27.6 -> 24.0 Improved
What 547 accepted papers report for this change
- 416 improved
- 28 no clear effect
- 14 mixed
- 89 hurt
- Lightweight Federated Incremental Learning via Decoupled Replay ICML 2026 · add mixup to uploaded features · CIFAR100 (α=1.0) · A(f): 33.20 -> 35.61Improved
- Quantifying and Optimizing Simplicity via Polynomial Representations ICML 2026 · add embedding mixup · GLUE RTE, MRPC, CoLA · accuracy / Matthews corr.: MRPC 86.74 -> 86.30; CoLA 62.31 -> 61.04; RTE unchanged 70.28Hurt
- Fix the Mind, Not the Move: Interpretable AI Assistance via Knowledge-Gap Localization ICML 2026 · add auxiliary mixup generalization loss · Breakfast · System recall: 0.221 -> 0.762Improved
See this change in detailMatched by the wording of each paper's own result, read by AI from the full text. Papers report gains more often than failures, so this shows what gets published, not the chance a change works.
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace BatchNorm with Fixup initialization | ResNet-110 · CIFAR-10 | test error (%): 6.61 -> 7.24 | Hurt |
| Replace Xavier initialization with Fixup initialization | ResNet-110 · CIFAR-10 | test error (%): 7.78 -> 7.24 | Improved |
| Add Mixup regularization to Fixup | ResNet-50 · ImageNet | test error (%): 27.6 -> 24.0 | Improved |
| Replace BatchNorm with Fixup and add Mixup | ResNet-50 · ImageNet | test error (%): 23.3 -> 24.0 | Hurt |
| Replace BatchNorm with Fixup and add Mixup | ResNet-101 · ImageNet | test error (%): 20.8 -> 21.4 | Hurt |
| Replace LayerNorm with Fixup | Transformer · IWSLT DE-EN | BLEU: 34.2 -> 34.5 | Improved |
| Replace LayerNorm with Fixup | Transformer · WMT EN-DE | BLEU: 29.3 -> 29.3 | No clear effect |
| Replace BatchNorm with Fixup and add Mixup + Cutout | WideResNet-40-10 · CIFAR-10 | test error (%): 2.5 -> 2.3 | Improved |
| Replace BatchNorm with Fixup and add Mixup + Cutout | WideResNet-16-12 · SVHN | test error (%): 1.4 -> 1.4 | No clear effect |
| Remove bias parameters (ablation) | Fixup ResNet-110 · CIFAR-10 | training accuracy | Hurt |
| Use 0.1x the suggested initialization scale and no bias parameters (ablation) | Fixup ResNet-110 · CIFAR-10 | training accuracy | Hurt |
| Use 10x the suggested initialization scale and no bias parameters (ablation) | Fixup ResNet-110 · CIFAR-10 | training accuracy | Hurt |
Closest papers by meaning
- Initialization and Regularization of Factorized Neural Layers (ICLR 2021 · similarity 0.88)
- Batch Normalization Biases Residual Blocks Towards the Identity Function in Deep Networks (NeurIPS 2020 · similarity 0.87)
- Four Things Everyone Should Know to Improve Batch Normalization (ICLR 2020 · similarity 0.87)
- IDInit: A Universal and Stable Initialization Method for Neural Network Training (ICLR 2025 · similarity 0.86)
- Mix-LN: Unleashing the Power of Deeper Layers by Combining Pre-LN and Post-LN (ICLR 2025 · similarity 0.86)
- Reducing Divergence in Batch Normalization for Domain Adaptation (AAAI 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
