Research map / Label-efficient and robust learning / Self-supervised and contrastive
Data-efficient and robust self-supervision
Improves self-supervised training with data selection, geometric harmonization and defenses against attacks.
122 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2020–2026) · an approach in Self-supervised and contrastive, keyword cluster “supervised ssl · representations · masked”
State History Benchmarks Tried Baselines Checklist Preprints Who Export
Where it stands
Leveling off Several papers from 2025–26 still report on 2 tracked benchmarks, but the best result has barely moved since 2024.
47 papers in 2023–24 and 48 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic held steady, from 14.5% to 15.0%.
Open questions the data points to
- Papers report MAE on ADE20k (mIoU) anywhere from 33.3 to 48.1: comparisons against it depend on whose number you take.
- Papers report SimSiam on ImageNet-100 (KNN accuracy (%)) anywhere from 62 to 71.21: comparisons against it depend on whose number you take.
What looks settled
- ImageNet-100 (top-1 accuracy): best up to 2024 86.72, best in 2025–26 80.9 from 3 papers, at most 0.2% better, if at all.
- CIFAR10 (linear accuracy): best up to 2024 93.96, best in 2025–26 93.53 from 2 papers, at most 0.2% better, if at all.
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2020 · Self-Supervised Relational Reasoning for Representation Learning (NeurIPS 2020 · 29 citations)
- 2021 · Barlow Twins: Self-Supervised Learning via Redundancy Reduction (ICML 2021 · 770 citations)
- 2022 · Self-supervised Learning is More Robust to Dataset Imbalance (ICLR 2022 · 64 citations)
- 2023 · Benchmarking Self-Supervised Learning on Diverse Pathology Datasets (CVPR 2023 · 179 citations)
- 2024 · Transcriptomics-guided Slide Representation Learning in Computational Pathology (CVPR 2024 · 46 citations)
- 2025 · Unsupervised Foundation Model-Agnostic Slide-Level Representation Learning (CVPR 2025 · 12 citations)
- 2026 · Self-Supervised Learning Based on Transformed Image Reconstruction for Equivariance-Coherent Feature Representation (AAAI 2026 · 1 citation)
Earliest papers here
- Self-Supervised Relational Reasoning for Representation Learning (NeurIPS 2020 · 29 citations)
- CompRess: Self-Supervised Learning by Compressing Representations (NeurIPS 2020 · 24 citations)
- Exploit Clues From Views: Self-Supervised and Regularized Learning for Multiview Object Recognition (CVPR 2020 · 6 citations)
Benchmark records
The best number each year that beat every earlier one, among papers of this approach reporting their own method. Numbers are extracted by AI from each paper's text; check the paper before citing.
ImageNet-100 · top-1 accuracy (higher is better)
6 papers, 2024–2025 · flat since 2024
| Year | Best so far | Set by |
|---|---|---|
| 2024 | 86.72 | Measuring Self-Supervised Representation Quality for Downstream Classification Using Discriminative Features (AAAI 2024 · 7 citations) |
ADE20k · mIoU (higher is better)
4 papers, 2023–2025 · one report since 2024, not better
| Year | Best so far | Set by |
|---|---|---|
| 2023 | 51.1 | Siamese Image Modeling for Self-Supervised Vision Representation Learning (CVPR 2023 · 87 citations) |
CIFAR10 · linear accuracy (higher is better)
4 papers, 2022–2025 · flat since 2024
| Year | Best so far | Set by |
|---|---|---|
| 2022 | 88.02 | Leverage Your Local and Global Representations: A New Self-Supervised Learning Strategy (CVPR 2022 · 24 citations) |
| 2024 | 93.96 | Self-Supervised Representation Learning with Meta Comprehensive Regularization (AAAI 2024 · 6 citations) |
What has been tried
85 papers here report 994 measured changes (a component added, removed or swapped, compared with a baseline): helped 62%, no clear effect 10%, hurt 24%, mixed 4%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| replace cross-modal contrastive alignment with expression reconstruction | Few-shot classification, larger k · Macro-AUC | Hurt | Transcriptomics-guided Slide Representation Learning in Computational Pathology (CVPR 2024 · 46 citations) | |
| Pretrain using three magnifications instead of only 0.5 MPP | CPTAC downstream tasks (average across models) · AUC | +1.73% AUC | Helped | Unsupervised Foundation Model-Agnostic Slide-Level Representation Learning (CVPR 2025 · 12 citations) |
| Remove the projector | CIFAR-100 · Top-1 accuracy | 91.64 -> 88.64 | Hurt | Preventing Dimensional Collapse in Self-Supervised Learning via Orthogonality Regularization (NeurIPS 2024 · 8 citations) |
| Select discriminative features from the middle of activation frequency rather than random features | ImageNet-1K · top-1 accuracy | Helped | Measuring Self-Supervised Representation Quality for Downstream Classification Using Discriminative Features (AAAI 2024 · 7 citations) | |
| Use all listed augmentations together rather than horizontal flip and rotate only | CIFAR-10 · linear accuracy | 92.64 -> 93.96 | Helped | Self-Supervised Representation Learning with Meta Comprehensive Regularization (AAAI 2024 · 6 citations) |
| replace Batch-wise Barlow Twins loss with InfoNCE loss | Kvasir · accuracy | 74.9 ± 0.6 -> 72.6 ± 0.4 | Hurt | Self-supervised Representation Learning from Random Data Projectors (ICLR 2024 · 3 citations) |
| remove the most memorized pretraining samples instead of the same number of random samples | CIFAR10, CIFAR100, and STL10 linear probing · linear probing accuracy | Hurt | Memorization in Self-Supervised Learning Improves Downstream Generalization (ICLR 2024 · 3 citations) | |
| replace representation distillation with KL-divergence | CIFAR-10, CIFAR-100; linear probing, public IID, Class and IID client distributions · Top-1 accuracy | 64.81 ± 0.02 -> 43.88 ± 0.19; 69.07 ± 0.04 -> 46.24 ± 0.08; 36.33 ± 0.01 -> 19.56 ± 0.18; 35.94 ± 0.02 -> 15.45 ± 0.05 | Hurt | Resource-Aware Federated Self-Supervised Learning with Global Class Representations (NeurIPS 2024 · 3 citations) |
| Add reconstruction loss to contrastive loss | GE · accuracy | 0.581 -> 0.694 | Helped | Representation Space Augmentation for Effective Self-Supervised Learning on Tabular Data (AAAI 2025 · 2 citations) |
| add the explicit alignment term | CIFAR-10 linear · accuracy | 92.42 -> 93.01 | Helped | Adv-SSL: Adversarial Self-Supervised Representation Learning with Theoretical Guarantees (NeurIPS 2025 · 2 citations) |
| Add the OSDT encoder to the neural autoencoder | Classification datasets, 400 labels · mean accuracy | 0.6815 -> 0.7124 | Helped | Hybrid Autoencoders for Tabular Data: Leveraging Model-Based Augmentation in Low-Label Settings (NeurIPS 2025 · 1 citation) |
| remove the regularization token [REG] | Jannis · representation collapse | Hurt | T-JEPA: Augmentation-Free Self-Supervised Learning for Tabular Data (ICLR 2025 · 1 citation) |
Baseline numbers that disagree
The same baseline on the same benchmark and metric, reported more than 5% apart by different papers here. Training budgets, data splits and reimplementations differ, so state which number you compare with.
| Baseline | Benchmark | Metric | Reported | Papers |
|---|---|---|---|---|
| MAE | ADE20k | mIoU | 33.3 – 48.1 | 2 papers, e.g. NeurIPS 2024 and CVPR 2023 |
| SimSiam | ImageNet-100 | KNN accuracy (%) | 62 – 71.21 | 2 papers, e.g. ICML 2025 and CVPR 2022 |
| SimSiam | CIFAR10 | KNN accuracy (%) | 80.48 – 88.1 | 2 papers, e.g. CVPR 2022 and ICML 2025 |
What reviewers will expect
Datasets and baselines named in the 27 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- CIFAR-10 · 7 of 27 papers
- CIFAR-100 · 7 of 27 papers
- ImageNet-100 · 6 of 27 papers
- ImageNet-1k · 6 of 27 papers
- Tiny ImageNet · 4 of 27 papers
- ImageNet · 3 of 27 papers
- TCGA-BRCA · 2 of 27 papers
- TCGA-LUAD · 2 of 27 papers
Baselines
- SimCLR · 10 of 27 papers
- BYOL · 9 of 27 papers
- SimSiam · 7 of 27 papers
- DINO · 6 of 27 papers
- VICReg · 6 of 27 papers
- Barlow Twins · 4 of 27 papers
- XGBoost · 3 of 27 papers
- CatBoost · 3 of 27 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 95 papers; from OpenAlex and the papers' affiliations).
- Meta · 8 papers
- Massachusetts Institute of Technology · 6 papers
- Google · 6 papers
- Beihang University · 4 papers
- New York University · 4 papers
- Chinese Academy of Sciences · 4 papers
- Shanghai Artificial Intelligence Laboratory · 3 papers
- University of Maryland, College Park · 3 papers
- University of Chinese Academy of Sciences · 3 papers
- Stanford University · 3 papers
Use it
Download BibTeX of 38 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Invariance-based self-supervised learning (168 papers)
- Masked autoencoder pretraining (48 papers)
- Self-supervised speech representations (24 papers)
Elsewhere in the topic
- Pretext-task visual pre-training (136 papers)
- Self-supervision in supervised pipelines (121 papers)
- Self-supervised neuroimaging representations (98 papers)
- Contrastive versus non-contrastive theory (74 papers)
- Inductive bias and minimal representations (66 papers)
- Self-supervised speech and brain decoding (56 papers)
- Empirical contrastive visual learning (38 papers)
- Representation evaluation and trade-offs (37 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).