Research map / Label-efficient and robust learning / Self-supervised and contrastive
Invariance-based self-supervised learning
Learns invariant representations through pretext invariance and causal or functional regularization.
168 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2018–2026) · an approach in Self-supervised and contrastive, keyword cluster “supervised ssl · representations · masked”
State History Tried Checklist Preprints Who Export
Where it stands
Not enough benchmark data Fewer than four papers report the same benchmark, so there is no record to follow.
53 papers in 2023–24 and 65 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic grew, from 16.4% to 20.2%.
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2018 · Trading robust representations for sample complexity through self-supervised visual experience (NeurIPS 2018 · 1 citation)
- 2019 · Revisiting Self-Supervised Visual Representation Learning (CVPR 2019 · 690 citations)
- 2020 · Bootstrap Your Own Latent - A New Approach to Self-Supervised Learning (NeurIPS 2020 · 3,384 citations)
- 2021 · SEED: Self-supervised Distillation For Visual Representation (ICLR 2021 · 71 citations)
- 2022 · VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning (ICLR 2022 · 335 citations)
- 2023 · Self-Supervised Learning From Images With a Joint-Embedding Predictive Architecture (CVPR 2023 · 399 citations)
- 2024 · SwitchTab: Switched Autoencoders Are Effective Tabular Learners (AAAI 2024 · 30 citations)
- 2025 · A Regularization-Guided Equivariant Approach for Image Restoration (CVPR 2025 · 3 citations)
- 2026 · CryoLVM: Self-supervised Learning from Cryo-EM Density Maps with Large Vision Models (ICLR 2026 · 1 citation)
Earliest papers here
- Trading robust representations for sample complexity through self-supervised visual experience (NeurIPS 2018 · 1 citation)
- Revisiting Self-Supervised Visual Representation Learning (CVPR 2019 · 690 citations)
- Learning Representations by Maximizing Mutual Information Across Views (NeurIPS 2019 · 671 citations)
What has been tried
111 papers here report 1,273 measured changes (a component added, removed or swapped, compared with a baseline): helped 63%, no clear effect 8%, hurt 22%, mixed 7%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| add switching reconstruction of swapped salient and mutual features | Bank (BK) · AUC | 0.918 -> 0.942 | Helped | SwitchTab: Switched Autoencoders Are Effective Tabular Learners (AAAI 2024 · 30 citations) |
| add local consistency and semantic relation to global consistency | RAF-DB · accuracy | 87.82 -> 88.72 | Helped | Self-Supervised Facial Representation Learning with Facial Region Awareness (CVPR 2024 · 25 citations) |
| add feature reconstruction warm-up to hyperedge filling without projection heads (V1 to V4) | node classification, 11 benchmark datasets · average ranking | 4.3 -> 2.5 | Helped | HypeBoy: Generative Self-Supervised Representation Learning on Hypergraphs (ICLR 2024 · 4 citations) |
| condition on generated representations instead of no condition | ImageNet 256×256 validation · FID | 14.23 -> 5.07 | Helped | Return of Unconditional Generation: A Self-supervised Representation Generation Method (NeurIPS 2024 · 4 citations) |
| add rank regularization to SimCLR | UTKFace (age), UTKFace (gender), CelebA (makeup) · bias-conflict accuracy | 36.4 -> 26.6; 56.3 -> 50.9; 46.9 -> 43.9 | Hurt | Self-supervised Debiasing Using Low Rank Regularization (CVPR 2024 · 3 citations) |
| Replace standard augmentations with only generative augmentation | ImageNet validation · Top-1 accuracy | +1.96 | Helped | Can Generative Models Improve Self-Supervised Representation Learning? (AAAI 2025 · 2 citations) |
| Add self-supervised transformation learning loss | Out-domain classification (11-dataset mean) · linear evaluation accuracy (%) | 48.49 -> 49.97 | Helped | Self-supervised Transformation Learning for Equivariant Representations (NeurIPS 2024 · 2 citations) |
| Increase encoder depth | Gaussian data with features sharing λ but differing in ρ · temporal spread between feature learning | Helped | How JEPA Avoids Noisy Features: The Implicit Bias of Deep Linear Self Distillation Networks (NeurIPS 2024 · 2 citations) | |
| Train the straightening objective on temporally shuffled frames | Sequential MNIST · visual-feature decoding performance | Hurt | Learning predictable and robust neural representations by straightening image sequences (NeurIPS 2024 · 2 citations) | |
| Replace the ResNet decoder with ViT | 5D all-in-one restoration · PSNR | 30.72 -> 30.88; 37.32 -> 37.74; 31.32 -> 31.37; 28.84 -> 28.93; 23.35 -> 23.22 | Mixed | Universal Image Restoration Pre-training via Degradation Classification (ICLR 2025 · 1 citation) |
| Normalize features and use a projection head | ImageNet-100 · top-1 kNN accuracy | 88.3 -> 89.1 | Helped | No Train, all Gain: Self-Supervised Gradients Improve Deep Frozen Representations (NeurIPS 2024 · 1 citation) |
| Increase K for the KNN context from 5 to 30 to 150 | abalone · Linear-probe prediction error | Mixed | Contextures: Representations from Contexts (ICML 2025 · 0 citations) |
What reviewers will expect
Datasets and baselines named in the 28 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- CIFAR100 · 6 of 28 papers
- ImageNet · 5 of 28 papers
- CIFAR10 · 5 of 28 papers
- ImageNet-1K · 4 of 28 papers
- Food101 · 2 of 28 papers
- iNaturalist 2018 · 2 of 28 papers
- Urban100 · 2 of 28 papers
- MNIST · 2 of 28 papers
Baselines
- DINO · 3 of 28 papers
- MoCo-v3 · 2 of 28 papers
- I-JEPA · 2 of 28 papers
- MAE · 2 of 28 papers
- Random · 2 of 28 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 118 papers; from OpenAlex and the papers' affiliations).
- New York University · 11 papers
- Meta · 8 papers
- Massachusetts Institute of Technology · 7 papers
- Mila · 5 papers
- Korea Advanced Institute of Science and Technology · 3 papers
- University of Maryland, College Park · 3 papers
- University of Montreal · 3 papers
- Microsoft · 3 papers
- Google · 3 papers
- University of Oslo · 3 papers
Use it
Download BibTeX of 40 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Data-efficient and robust self-supervision (122 papers)
- Masked autoencoder pretraining (48 papers)
- Self-supervised speech representations (24 papers)
Elsewhere in the topic
- Pretext-task visual pre-training (136 papers)
- Self-supervision in supervised pipelines (121 papers)
- Self-supervised neuroimaging representations (98 papers)
- Contrastive versus non-contrastive theory (74 papers)
- Inductive bias and minimal representations (66 papers)
- Self-supervised speech and brain decoding (56 papers)
- Empirical contrastive visual learning (38 papers)
- Representation evaluation and trade-offs (37 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).