Research map / Label-efficient and robust learning / Self-supervised and contrastive
On the Efficacy of Small Self-Supervised Contrastive Models without Distillation Signals
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Combine lower temperature, wider projector, stronger augmentation, and better initialization (ablation) | MobileNetV3_large MoCoV2 · ImageNet1k | linear evaluation accuracy: 33.5 -> 46.0 | Helped |
| Train for 800 rather than 200 epochs with the improved recipe (ablation) | MobileNetV3_large · ImageNet1k | linear evaluation accuracy: 42.6 -> 47.9 | Helped |
| Use lower temperature (0.05 rather than 0.2) (ablation) | MobileNetV3_large MoCoV2 · ImageNet1k | linear evaluation accuracy: 32.2 -> 33.5 | Helped |
| Reduce negative sample size from 65536 to 16384 (ablation) | MobileNetV3_large MoCoV2 · ImageNet1k | linear evaluation accuracy: 33.5 -> 34.2 | No clear effect |
| Strengthen color augmentation to the best tested scheme (ablation) | MobileNetV3_large MoCoV2 · ImageNet1k | linear evaluation accuracy: 33.5 -> 36.6 | Helped |
| Initialize from a better SEED checkpoint (ablation) | MobileNetV3_large MoCoV2 · ImageNet1k | linear evaluation accuracy: 33.5 -> 39.4 | Helped |
| Widen the projector intermediate layer (ablation) | MobileNetV3_large MoCoV2 · ImageNet1k | linear evaluation accuracy: 33.5 -> 33.9 | Helped |
| Add a projector layer (ablation) | MobileNetV3_large MoCoV2 · ImageNet1k | linear evaluation accuracy: 33.5 -> 29.0 | Hurt |
| Add dropout (p=0.5) to the projector (ablation) | MobileNetV3_large MoCoV2 · ImageNet1k | linear evaluation accuracy: 33.5 -> 28.9 | Hurt |
| Train longer (800 rather than 200 epochs) (ablation) | MobileNetV3 architectures with the improved recipe · ImageNet1k | linear evaluation accuracy: 48.7 vs 47.9; 41.3 vs 41.0 | Mixed |
| Apply the combined improved recipe instead of baseline MoCoV2 | Five small architectures · ImageNet1k | linear evaluation accuracy: 32.2 -> 47.9; 26.8 -> 41.0; 42.1 -> 55.9; 51.1 -> 55.7; 23.4 -> 38.6 | Mixed |
Closest papers by meaning
- SEED: Self-supervised Distillation For Visual Representation (ICLR 2021 · similarity 0.90)
- A theoretical study of inductive biases in contrastive learning (ICLR 2023 · similarity 0.89)
- Self-Supervised Contrastive Learning is Approximately Supervised Contrastive Learning (NeurIPS 2025 · similarity 0.88)
- Boost Self-Supervised Dataset Distillation via Parameterization, Predefined Augmentation, and Approximation (ICLR 2025 · similarity 0.88)
- Dataset Distillation via Knowledge Distillation: Towards Efficient Self-Supervised Pre-training of Deep Networks (ICLR 2025 · similarity 0.87)
- M2SD:Multiple Mixing Self-Distillation for Few-Shot Class-Incremental Learning (AAAI 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.