Research map / Label-efficient and robust learning / Semi-supervised learning
Exponential Moving Average Normalization for Self-Supervised and Semi-Supervised Learning
Check your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace the teacher's batch normalization with EMAN | FixMatch · ImageNet, 10% labels | top-1 accuracy: 67.1 -> 69.2 | Helped |
| Replace the teacher's ShuffleBN with EMAN | MoCo · ImageNet, 10% labels | accuracy: 54.4 -> 55.8 | Helped |
| Replace the teacher's SyncBN with EMAN | BYOL · ImageNet, 10% labels | accuracy: 55.4 -> 56.2 | Helped |
| Use EMAN in MoCo pretraining | MoCo · ImageNet, 1% labels, linear evaluation | top-1 accuracy: 50.9 -> 57.4 | Helped |
| Use EMAN in FixMatch, initialize from MoCo-EMAN, and train 3× | FixMatch · ImageNet, 1% labels | top-1 accuracy: 63.0 | Helped |
| Set EMAN momentum to 0.9 (ablation) | MoCo · ImageNet, 10% labels | accuracy: 55.8 -> 51.2 | Hurt |
| Set EMAN momentum to 0.9 (ablation) | FixMatch · ImageNet, 10% labels | accuracy: 69.2 -> 69.0 | No clear effect |
| Set EMAN momentum to 0.99999 (ablation) | FixMatch · ImageNet, 10% labels | accuracy: 69.2 -> 54.6 | Hurt |
| Use teacher proxy statistics with alpha 0.9 instead of EMAN (ablation) | FixMatch · ImageNet, 10% labels | accuracy: 69.2 -> 61.4 | Hurt |
| Copy student proxy statistics to the teacher with alpha 0.9 instead of EMAN (ablation) | FixMatch · ImageNet, 10% labels | accuracy: 69.2 -> 68.6 | No clear effect |
| Use teacher proxy statistics with alpha 0.999 instead of EMAN (ablation) | FixMatch · ImageNet, 10% labels | accuracy: 69.2 -> 60.9 | Hurt |
| Copy student proxy statistics to the teacher with alpha 0.999 instead of EMAN (ablation) | BYOL · ImageNet, 10% labels | accuracy: 56.2 -> 56.2 | No clear effect |
Closest papers by meaning
- Iterative Normalization: Beyond Standardization Towards Efficient Whitening (CVPR 2019 · similarity 0.86)
- Online Normalization for Training Neural Networks (NeurIPS 2019 · similarity 0.84)
- Barely-Supervised Learning: Semi-supervised Learning with Very Few Labeled Images (AAAI 2022 · similarity 0.84)
- Normalizing the Normalizers: Comparing and Extending Network Normalization Schemes (ICLR 2017 · similarity 0.84)
- Big Self-Supervised Models are Strong Semi-Supervised Learners (NeurIPS 2020 · similarity 0.84)
- Adam with model exponential moving average is effective for nonconvex optimization (NeurIPS 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.