Research map / Label-efficient and robust learning / Self-supervised and contrastive
Revisiting Weakly Supervised Pre-Training of Visual Perception Models
Check your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Pre-train on hashtag-labeled IG 3.6B images rather than supervised JFT 300M | ViT H/14 · ImageNet-1k finetuning | accuracy: 88.6 vs 88.6 | Mixed |
| Use hashtag-labeled IG 3.6B pre-training | ViT H/14 · CUB-2011 finetuning | accuracy: 91.7 vs 91.3 | Helped |
| Add zero-shot transfer initialization before finetuning | RegNetY 128GF · ImageNet-1k, 1% labels | accuracy: 79.2 -> 82.0 | Helped |
| Add head-tail sampling with ratio 0.7 (ablation) | Hashtag pre-training with ResNeXt-101 32x8d · ImageNet-1k linear evaluation | accuracy: 74.9 -> 76.6 | Helped |
| Use head-tail sampling ratio 0.5 instead of the baseline collection (ablation) | Hashtag pre-training with ResNeXt-101 32x8d · ImageNet-1k linear evaluation | accuracy: 74.9 -> 76.3 | Helped |
| Use excessive tail oversampling with ratio 0.3 (ablation) | Hashtag pre-training with ResNeXt-101 32x8d · ImageNet-1k linear evaluation | accuracy: 74.9 -> 75.5 | Helped |
| Increase hashtag vocabulary from 17K to 27K (ablation) | Hashtag pre-training with ResNeXt-101 32x8d · ImageNet-1k linear evaluation | accuracy: 76.6 -> 77.0 | Helped |
| Add Platt scaling to correct the mismatch between hashtag and target-class distributions (ablation) | RegNetY 128GF zero-shot transfer · ImageNet-1k | accuracy: 65.1 -> 75.3 | Helped |
| Increase the model scale from RegNetY 16GF to RegNetY 128GF (ablation) | SWAG RegNetY · ImageNet-1k finetuning | accuracy: 86.0 -> 88.2 | Helped |
| Use larger models and longer pre-training | SWAG RegNetY · ImageNet-1k linear evaluation | accuracy | Helped |
| Increase dataset size while holding total training samples fixed (ablation) | SWAG pre-training · ImageNet-1k linear evaluation | accuracy: 81.5; 81.5; 81.4; 81.3 | No clear effect |
| Use exponential moving averages, mixup, label smoothing, AutoAugment, or stochastic depth during pre-training (ablation) | SWAG pre-training · ImageNet transfer | accuracy | No clear effect |
Closest papers by meaning
- Attend in Groups: A Weakly-Supervised Deep Learning Framework for Learning From Web Data (CVPR 2017 · similarity 0.88)
- Hard Mixtures of Experts for Large Scale Weakly Supervised Vision (CVPR 2017 · similarity 0.88)
- ClusterFit: Improving Generalization of Visual Representations (CVPR 2020 · similarity 0.88)
- Separating Self-Expression and Visual Content in Hashtag Supervision (CVPR 2018 · similarity 0.87)
- W2F: A Weakly-Supervised to Fully-Supervised Framework for Object Detection (CVPR 2018 · similarity 0.87)
- Beyond Random Augmentations: Pretraining with Hard Views (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.