Research map / Label-efficient and robust learning / Semi-supervised learning
Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace prediction-averaged or shared-weight targets with an EMA-weighted teacher | Mean Teacher versus Pi model · SVHN, 250 labels | error rate (%): 9.69 \pm 0.92 -> 4.35 \pm 0.50 | Helped |
| Replace prediction-averaged or shared-weight targets with an EMA-weighted teacher | Mean Teacher versus Pi model · CIFAR-10, 1000 labels | error rate (%): 27.36 \pm 1.20 -> 21.55 \pm 1.48 | Helped |
| Use an EMA-weighted teacher instead of the Pi model | Mean Teacher · SVHN, all labels | error rate (%): 2.50 \pm 0.07 -> 2.50 \pm 0.05 | No clear effect |
| Use Mean Teacher rather than Temporal Ensembling | Semi-supervised learning · SVHN, 500 labels | error rate (%): 5.12 \pm 0.13 -> 4.18 \pm 0.27 | Helped |
| Use Mean Teacher rather than VAT+EntMin | Semi-supervised learning · SVHN, 1000 labels | error rate (%): 3.86 -> 3.95 \pm 0.19 | Hurt |
| Add a ResNet architecture to Mean Teacher instead of using the ConvNet | Mean Teacher · CIFAR-10, 4000 labels | error rate (%): 12.31 \pm 0.28 -> 6.28 \pm 0.15 | Helped |
| Add input augmentation (ablation) | Mean Teacher · SVHN, 250 labels | error rate (%): 5.85 \pm 0.62 -> 4.35 \pm 0.50 | Helped |
| Add input augmentation (ablation) | Mean Teacher · CIFAR-10, 4000 labels | error rate (%): 17.74 \pm 0.30 -> 12.31 \pm 0.28 | Helped |
| Remove input noise while using input augmentation (ablation) | Mean Teacher · SVHN, 250 labels | validation error | No clear effect |
| Add dropout (ablation) | Mean Teacher · SVHN, 250 labels | validation error | Helped |
| Add dropout to the teacher side when the student already has dropout (ablation) | Mean Teacher · SVHN, 250 labels | validation error | Helped |
| Use MSE rather than KL-divergence for the consistency cost (ablation) | Mean Teacher · SVHN, 250 labels | validation error | Helped |
Closest papers by meaning
- Temporal Ensembling for Semi-Supervised Learning (ICLR 2017 · similarity 0.89)
- Fidelity-Weighted Learning (ICLR 2018 · similarity 0.86)
- Revisiting Knowledge Distillation via Label Smoothing Regularization (CVPR 2020 · similarity 0.86)
- Perturbed and Strict Mean Teachers for Semi-Supervised Semantic Segmentation (CVPR 2022 · similarity 0.85)
- Smooth Neighbors on Teacher Graphs for Semi-Supervised Learning (CVPR 2018 · similarity 0.84)
- MentorNet: Learning Data-Driven Curriculum for Very Deep Neural Networks on Corrupted Labels (ICML 2018 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.