Research map / Label-efficient and robust learning / Semi-supervised learning
Big Self-Supervised Models are Strong Semi-Supervised Learners
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use self-supervised pretraining, supervised fine-tuning, and unlabeled-data distillation | ResNet-50 · ImageNet 1% labels | top-1 accuracy: 73.9 | Helped |
| Use self-supervised pretraining, supervised fine-tuning, and unlabeled-data distillation | ResNet-50 · ImageNet 10% labels | top-1 accuracy: 77.5 | Helped |
| Increase model scale from ResNet-50 to ResNet-152 (3×+SK) | SimCLRv2 fine-tuning · ImageNet 1% labels | top-1 accuracy: 57.9 -> 74.9 | Helped |
| Add selective kernels (SK) (ablation) | SimCLRv2 ResNet-50, width 1× · ImageNet 1% labels | top-1 accuracy: 57.9 -> 64.5 | Helped |
| Add a third projection-head layer and fine-tune from its first layer (ablation) | SimCLRv2 · ImageNet 1% labels | top-1 accuracy: 14% relative improvement | Helped |
| Add MoCo-style memory (ablation) | SimCLRv2 · ImageNet linear evaluation and 1% fine-tuning | top-1 accuracy: ∼1% | Helped |
| Distill using teacher predictions on unlabeled examples (ablation) | ResNet-50 student · ImageNet 1% labels | top-1 accuracy: 12.3 -> 68.9 | Helped |
| Distill using teacher predictions on unlabeled examples (ablation) | ResNet-50 student · ImageNet 10% labels | top-1 accuracy: 52.0 -> 74.3 | Helped |
| Add ground-truth label loss alongside distillation loss (ablation) | ResNet-50 student · ImageNet 1% labels | top-1 accuracy: 68.9 -> 69.0 | No clear effect |
| Add ground-truth label loss alongside distillation loss (ablation) | ResNet-50 student · ImageNet 10% labels | top-1 accuracy: 74.3 -> 75.1 | Helped |
| Include unlabeled examples in the distillation-loss training set (ablation) | ResNet-50 student · ImageNet 1% labels | top-1 accuracy: 23.6 -> 69.0 | Helped |
| Use distillation temperature 2.0 instead of 0.1 or 1.0 (ablation) | SimCLRv2 distillation · ImageNet 1% and 10% labels | top-1 accuracy | Hurt |
Closest papers by meaning
- Learning to Self-Train for Semi-Supervised Few-Shot Classification (NeurIPS 2019 · similarity 0.90)
- Regularization With Stochastic Transformations and Perturbations for Deep Semi-Supervised Learning (NeurIPS 2016 · similarity 0.89)
- Semi-Supervised Learning Made Simple With Self-Supervised Clustering (CVPR 2023 · similarity 0.89)
- Barely-Supervised Learning: Semi-supervised Learning with Very Few Labeled Images (AAAI 2022 · similarity 0.89)
- Revisiting Semi-Supervised Learning in the Era of Foundation Models (NeurIPS 2025 · similarity 0.89)
- CGMatch: A Different Perspective of Semi-supervised Learning (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.