Research map / Label-efficient and robust learning / Self-supervised and contrastive
Divide and Contrast: Self-Supervised Learning From Uncurated Data
Yonglong Tian, Olivier J. Hénaff, Aäron van den Oord
This paper tried a common change: Train longer
train longer (1000 to 3000 epochs) with MoCLR · ImageNet linear (pretrained on ImageNet) · top-1 accuracy (%): 74.3 -> 74.5 No clear effect
What 387 accepted papers report for this change
- 211 improved
- 62 no clear effect
- 30 mixed
- 84 hurt
- Lightweight Federated Incremental Learning via Decoupled Replay ICML 2026 · increase local training epochs from 5 to 40 · CIFAR10 (α=1.0) · test accuracyImproved
- When Do Hallucinations Arise? A Graph Perspective on the Evolution of Path Reuse and Path Compression ICML 2026 · train longer on the full path search space (overfitting stage) · SBM graph, 10 communities, pin=0.6 · accuracy: 1.0 -> 0.98Hurt
- Causes and Consequences of Representational Similarity in Machine Learning Models ICML 2026 · train longer on LLM continued pretraining to test insufficient adaptation · perplexity across overlap settings · perplexityImproved
See this change in detailMatched by the wording of each paper's own result, read by AI from the full text. Papers report gains more often than failures, so this shows what gets published, not the chance a change works.
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add Divide and Contrast (cluster, local experts, distillation) to MoCLR | MoCLR, ResNet-50 · ImageNet linear (pretrained on JFT-300M, 3000 epochs) | top-1 accuracy (%): 67.4 -> 69.8 | Improved |
| add Divide and Contrast to MoCLR | MoCLR, ResNet-50 · ImageNet linear (pretrained on YFCC100M, 3000 epochs) | top-1 accuracy (%): 65.7 -> 67.8 | Improved |
| add Divide and Contrast to MoCLR | MoCLR, ResNet-200x2 · ImageNet linear (pretrained on JFT-300M, 3000 epochs) | top-1 accuracy (%): 74.2 -> 77.3 | Improved |
| add Divide and Contrast to MoCLR | MoCLR, ResNet-50 · ImageNet linear (pretrained on ImageNet, 3000 epochs) | top-1 accuracy (%): 74.5 -> 75.8 | Improved |
| train longer (1000 to 3000 epochs) with MoCLR | MoCLR, ResNet-50 · ImageNet linear (pretrained on ImageNet) | top-1 accuracy (%): 74.3 -> 74.5 | No clear effect |
| train longer (1000 to 3000 epochs) with BYOL | BYOL, ResNet-50 · ImageNet linear (pretrained on ImageNet) | top-1 accuracy (%): 74.3 -> 73.9 | Hurt |
| add DnC to MoCLR for semi-supervised learning | MoCLR, ResNet-50 · ImageNet semi-supervised (1% labels) | top-1 accuracy (%): 53.0 -> 59.9 | Improved |
| add DnC to MoCLR for fine-grained linear transfer | MoCLR-5k, ResNet-50 · 12-dataset linear transfer, average (pretrained on JFT-300M) | average accuracy (%): 71.7 -> 76.3 | Improved |
| add DnC to MoCLR for depth estimation (YFCC100M) | MoCLR-5k · NYU v2 depth estimation | delta < 1.25 accuracy (%): 86.7 -> 86.2 | No clear effect |
| train experts on the full dataset instead of local clusters (ablation) | DnC · ImageNet linear (DnC, 3000 epochs) | top-1 accuracy (%): 74.3 -> 75.8 | Improved |
| partition by clustering instead of random subsets (ablation) | DnC · ImageNet linear (DnC, 3000 epochs) | top-1 accuracy (%): 73.1 -> 75.8 | Improved |
| distill from both base model and local experts instead of base only (ablation) | DnC · ImageNet linear (DnC, 3000 epochs) | top-1 accuracy (%): 74.5 -> 75.8 | Improved |
Closest papers by meaning
- Self-Contrastive Learning: Single-Viewed Supervised Contrastive Framework Using Sub-network (AAAI 2023 · similarity 0.91)
- Self-Supervised Contrastive Learning is Approximately Supervised Contrastive Learning (NeurIPS 2025 · similarity 0.89)
- SELECT: A Large-Scale Benchmark of Data Curation Strategies for Image Classification (NeurIPS 2024 · similarity 0.88)
- A Tale of Two Classes: Adapting Supervised Contrastive Learning to Binary Imbalanced Datasets (CVPR 2025 · similarity 0.88)
- $\mathbb{X}$-Sample Contrastive Loss: Improving Contrastive Learning with Sample Similarity Graphs (ICLR 2025 · similarity 0.87)
- Collapse-Proof Non-Contrastive Self-Supervised Learning (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
