Research map / Optimization / Optimizers
Exploiting Invariance in Training Deep Neural Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace SyncBN with ND++ feature transforms | ImageNet ResNet-50 · ImageNet val | top-1 accuracy: 76.25 -> 77.95 | Helped |
| Replace the normalization baseline with ND++ | Mask R-CNN, Detectron2 · COCO 2017 | bounding box AP: 38.96 -> 39.91 | Helped |
| Replace the normalization baseline with ND++ | Mask R-CNN, Detectron2 · COCO 2017 | mask AP: 35.93 -> 36.87 | Helped |
| Use ND++ instead of SyncBN | DeepLabv3-ResNet-50 trained from scratch · Cityscapes val | mIoU: 69.30 -> 72.69 | Helped |
| Use ND++ instead of the baseline Transformer | Transformer language model · WikiText-103 | perplexity: 31.48 -> 29.54 | Helped |
| Add scale invariance to ND | ViT-B/16 · CIFAR-10 | accuracy: 70% -> 80% | Helped |
| Add scale invariance (ablation) | ND++ · Cityscapes val | mIoU: 72.69 -> 72.69 | No clear effect |
| Add cross-GPU synchronization (ablation) | ND++ on DeepLabv3-ResNet-50 · Cityscapes val | mIoU: 69.82 -> 72.69 | Helped |
| Add scale invariance (ablation) | ND++ on ImageNet ResNet-50 · ImageNet val | top-1 accuracy: 77.80% -> 77.95% | No clear effect |
| Train Mask R-CNN from scratch with learning rate 0.1 instead of 0.02 (ablation) | ND++ · COCO 2017 | bounding box AP: 37.81 -> 38.86 | Helped |
| Train with initial learning rate 1.0 and no momentum | ND++ · ImageNet | top-1 accuracy | Helped |
| Increase batch size to 2048 | ND++ ImageNet training · ImageNet | training iterations: one-eighth the number of iterations | Helped |
Closest papers by meaning
- Learning Invariances in Neural Networks from Training Data (NeurIPS 2020 · similarity 0.92)
- In What Ways Are Deep Neural Networks Invariant and How Should We Measure This? (NeurIPS 2022 · similarity 0.90)
- On Making $SE(2)$-Invariant Networks Optimal (NeurIPS 2026 · similarity 0.87)
- Continuous Invariance Learning (ICLR 2024 · similarity 0.87)
- Rethinking Invariance Regularization in Adversarial Training to Improve Robustness-Accuracy Trade-off (ICLR 2025 · similarity 0.87)
- Structured Initialization for Vision Transformers (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.