Research map / Label-efficient and robust learning / Self-supervised and contrastive
Effective Self-supervised Pre-training on Low-compute Networks without Distillation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use capacity-adapted views with MoCo-v2* | MoCo-v2* on MobileNetV2 · ImageNet-1K linear evaluation | Top-1(%): 60.6 -> 61.6 | Helped |
| Use capacity-adapted views with SwAV | SwAV on MobileNetV2 · ImageNet-1K linear evaluation | Top-1(%): 65.2 -> 67.3 | Helped |
| Use capacity-adapted views with DINO | DINO on MobileNetV2 · ImageNet-1K linear evaluation | Top-1(%): 66.2 -> 68.3 | Helped |
| Use capacity-adapted view sampling | DINO on ResNet18 · ImageNet-1K linear evaluation | Top-1(%): 62.2 -> 65.7 | Helped |
| Use capacity-adapted view sampling | DINO on ResNet34 · ImageNet-1K linear evaluation | Top-1(%): 67.7 -> 69.7 | Helped |
| Use capacity-adapted view sampling | DINO on ResNet50 · ImageNet-1K linear evaluation | Top-1(%): 73.4 -> 73.4 | No clear effect |
| Use the proposed training recipe | DINO on MobileNetV2 · ImageNet-1K linear evaluation | Top-1(%): +2.1 | Helped |
| Use the proposed training recipe | DINO on MobileNetV2 · COCO object detection, MobileNetV2, 1x | AP^bb: 30.7 -> 32.4 | Helped |
| Extend proposed pretraining from 200 to 400 epochs | Ours on ResNet18 · ImageNet-1K linear evaluation | Top-1: 65.7 -> 66.8 | Helped |
| Change crop-scale ranges to [0.3, 0.4] (ablation) | DINO on MobileNetV2 · ImageNet-1K validation, linear accuracy | Top-1(%): 66.2 -> 67.0 | Helped |
| Change crop scales and align global/local pixel scales (ablation) | DINO on MobileNetV2 · ImageNet-1K validation, linear accuracy | Top-1(%): 67.0 -> 67.9 | Helped |
| Add loss re-balancing between global-global and global-local pairs (ablation) | DINO on MobileNetV2 · ImageNet-1K validation, linear accuracy | Top-1(%): +2.1 | Helped |
Closest papers by meaning
- DATA: Domain-Aware and Task-Aware Self-Supervised Learning (CVPR 2022 · similarity 0.90)
- Dataset Distillation via Knowledge Distillation: Towards Efficient Self-Supervised Pre-training of Deep Networks (ICLR 2025 · similarity 0.89)
- A Closer Look at Self-Supervised Lightweight Vision Transformers (ICML 2023 · similarity 0.88)
- Beyond Random Augmentations: Pretraining with Hard Views (ICLR 2025 · similarity 0.87)
- Harnessing small projectors and multiple views for efficient vision pretraining (NeurIPS 2024 · similarity 0.86)
- Probing the Mid-level Vision Capabilities of Self-Supervised Learning (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.