Research map / Label-efficient and robust learning / Self-supervised and contrastive
ISD: Self-Supervised Learning by Iterative Similarity Distillation
Ajinkya Tejankar, Soroush Abbasi Koohpayegani, Vipin Pillai, Paolo Favaro, Hamed Pirsiavash
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use weak augmentation for the student instead of strong (teacher strong) (ablation) | ISD · ImageNet NN, ResNet-50, 200 ep | NN accuracy: 58.0 -> 59.5 | Improved |
| use weak augmentation for the student instead of strong (teacher strong) (ablation) | ISD · ImageNet 20-NN, ResNet-50, 200 ep | 20-NN accuracy: 61.2 -> 62.2 | Improved |
| increase memory bank from 128K to 512K (ablation) | ISD · ImageNet NN, ResNet-50 | NN accuracy: 59.2 -> 59.0 | No clear effect |
| increase memory bank from 128K to 1024K (ablation) | ISD · ImageNet NN, ResNet-50 | NN accuracy: 59.2 -> 59.5 | No clear effect |
| increase memory bank from 128K to 1024K (ablation) | ISD · ImageNet 20-NN, ResNet-50 | 20-NN accuracy: 62.0 -> 62.2 | No clear effect |
| set temperature to 0.02 versus 0.003 (ablation) | ISD · ImageNet NN, ResNet-18, 130 ep | NN accuracy: 37.2 -> 39.7 | Improved |
| raise temperature from 0.02 to 0.06 (ablation) | ISD · ImageNet NN, ResNet-18, 130 ep | NN accuracy: 39.7 -> 32.5 | Hurt |
| tune BYOL learning rate to 0.05 versus 0.01 (ablation) | BYOL ResNet-18 · ImageNet NN, ResNet-18, 200 ep | NN accuracy: 37.3 -> 40.0 | Improved |
| replace MoCo hard contrastive labels with ISD soft similarity distillation | MoCo · ImageNet linear, ResNet-18, 200 ep | top-1 linear accuracy: 51.0 -> 53.8 | Improved |
| replace MoCo with ISD on unbalanced data, all 38 categories | MoCo · Unbalanced ImageNet-38, dataset D3 | NN accuracy: 59.4 -> 61.9 | Improved |
| replace MoCo with ISD on unbalanced data, rare categories only | MoCo · Unbalanced ImageNet-38, 30 rare categories, dataset D6 | NN accuracy: +3.8 | Improved |
| replace MoCo with ISD on unbalanced data, dataset D10 rare categories | MoCo · Unbalanced ImageNet-38, 30 rare categories, D10 | NN accuracy: 59.8 -> 51.5 | Hurt |
Closest papers by meaning
- Data-Efficient Contrastive Self-supervised Learning: Most Beneficial Examples for Supervised Learning Contribute the Least (ICML 2023 · similarity 0.90)
- On the duality between contrastive and non-contrastive self-supervised learning (ICLR 2023 · similarity 0.89)
- Divide and Contrast: Self-Supervised Learning From Uncurated Data (ICCV 2021 · similarity 0.89)
- Self-Supervised Contrastive Learning is Approximately Supervised Contrastive Learning (NeurIPS 2025 · similarity 0.88)
- Weakly-Supervised Contrastive Learning for Imprecise Class Labels (ICML 2025 · similarity 0.88)
- $\mathbb{X}$-Sample Contrastive Loss: Improving Contrastive Learning with Sample Similarity Graphs (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
