Research map / Label-efficient and robust learning / Self-supervised and contrastive
Boost Supervised Pretraining for Visual Transfer Learning: Implications of Self-Supervised Contrastive Representation Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Combine CAM-guided activation suppression with contrastive learning (ablation) | Supervised pretraining (CAMtrast) · miniImageNet, 5-way few-shot | accuracy: 33.46 -> 39.39 (1-shot); 45.30 -> 54.22 (5-shot) | Helped |
| Add CAM-guided activation suppression (ablation) | Supervised pretraining · miniImageNet, 5-way few-shot | accuracy: 33.46 -> 34.39 (1-shot); 45.30 -> 47.85 (5-shot) | Helped |
| Add contrastive learning with MoCo v2 (ablation) | Supervised pretraining · miniImageNet, 5-way few-shot | accuracy: 33.46 -> 35.38 (1-shot); 45.30 -> 48.68 (5-shot) | Helped |
| Add contrastive learning with SimSiam (ablation) | Supervised pretraining · miniImageNet, 5-way few-shot | accuracy: 33.46 -> 36.24 (1-shot); 45.30 -> 49.99 (5-shot) | Helped |
| Combine CAM-guided activation suppression and contrastive learning (ablation) | Supervised pretraining · miniImageNet, 5-way few-shot | representation entropy: 3.56 -> 4.47 | Helped |
| Combine CAM-guided activation suppression and contrastive learning (ablation) | Supervised pretraining · miniImageNet, 5-way few-shot | uniformity: 7.49 -> 14.37 | Helped |
| Use CAMtrast instead of supervised pretraining | ResNet-50 transfer to standard few-shot classification · miniImageNet and tieredImageNet | accuracy: 55.31±0.78 -> 58.42±0.78 (miniImageNet 1-shot); 61.06±0.84 -> 62.90±0.84 (tieredImageNet 1-shot) | Helped |
| Use CAMtrast (MoCo v2) instead of supervised pretraining | ResNet-50 transfer to object detection · PASCAL VOC07 test | AP: 70.21 -> 72.16 | Helped |
| Use CAMtrast (MoCo v2) instead of supervised pretraining | ResNet-50 transfer to semantic segmentation · Cityscapes validation | mIoU: 68.26 -> 74.01 | Helped |
| Reduce the number of supervised pretraining classes while holding total images at 4,800 | Supervised pretraining · miniImageNet, 5-way few-shot | accuracy: The paper reports a degradation of about 14–20% as classes decrease from 64 to 8. | Hurt |
| Reduce pretraining images per class while holding classes at 64 | Supervised pretraining · miniImageNet, 5-way few-shot | accuracy: 55.31 -> 43.12 (1-shot); 73.73 -> 60.36 (5-shot) | Hurt |
| Use fine-grained class labels rather than coarse category labels | Supervised pretraining · tieredImageNet, 5-way few-shot | accuracy: The paper reports absolute margins of about 0.31–7.79%. | Helped |
Closest papers by meaning
- When Does Contrastive Visual Representation Learning Work? (CVPR 2022 · similarity 0.91)
- Revisiting the Transferability of Supervised Pretraining: An MLP Perspective (CVPR 2022 · similarity 0.90)
- What Makes Instance Discrimination Good for Transfer Learning? (ICLR 2021 · similarity 0.90)
- Aligning Pretraining for Detection via Object-Level Contrastive Learning (NeurIPS 2021 · similarity 0.89)
- On the Surprising Effectiveness of Attention Transfer for Vision Transformers (NeurIPS 2024 · similarity 0.88)
- Is ImageNet worth 1 video? Learning strong image encoders from 1 long unlabelled video (ICLR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.