Research map / Label-efficient and robust learning / Self-supervised and contrastive
Self-supervision in supervised pipelines
Adds self-supervised objectives to supervised classifiers and other problem domains.
121 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2016–2026) · an approach in Self-supervised and contrastive, keyword cluster “depth · self supervision · pre”
State History Benchmarks Tried Checklist Preprints Who Export
Where it stands
Quiet None of the tracked benchmarks has a report from 2025–26.
35 papers in 2023–24 and 29 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic shrank, from 10.8% to 9.0%.
No longer reported by papers here: CIFAR-10 (ACC, last in 2023).
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2016 · Augmenting Supervised Neural Networks with Unsupervised Objectives for Large-scale Image Classification (ICML 2016 · 97 citations)
- 2017 · Adversarial Feature Learning (ICLR 2017 · 689 citations)
- 2018 · Self-Supervised Adversarial Hashing Networks for Cross-Modal Retrieval (CVPR 2018 · 495 citations)
- 2019 · Evaluating Protein Transfer Learning with TAPE (NeurIPS 2019 · 703 citations)
- 2020 · Adaptive Dilated Network With Self-Correction Supervision for Counting (CVPR 2020 · 187 citations)
- 2021 · TabNet: Attentive Interpretable Tabular Learning (AAAI 2021 · 1,769 citations)
- 2022 · Self-Supervised Learning of Adversarial Example: Towards Good Generalizations for Deepfake Detection (CVPR 2022 · 293 citations)
- 2023 · Spatio-Temporal Self-Supervised Learning for Traffic Flow Prediction (AAAI 2023 · 279 citations)
- 2024 · Parameter Efficient Self-Supervised Geospatial Domain Adaptation (CVPR 2024 · 20 citations)
- 2025 · Multi-aspect Self-guided Deep Information Bottleneck for Multi-modal Clustering (AAAI 2025 · 9 citations)
- 2026 · One protein is all you need (ICLR 2026 · 2 citations)
Earliest papers here
- Augmenting Supervised Neural Networks with Unsupervised Objectives for Large-scale Image Classification (ICML 2016 · 97 citations)
- Learning Bound for Parameter Transfer Learning (NeurIPS 2016 · 6 citations)
- Adversarial Feature Learning (ICLR 2017 · 689 citations)
Benchmark records
The best number each year that beat every earlier one, among papers of this approach reporting their own method. Numbers are extracted by AI from each paper's text; check the paper before citing.
CIFAR-10 · ACC (higher is better)
5 papers, 2019–2023 · not reported since 2023
| Year | Best so far | Set by |
|---|---|---|
| 2019 | 93.49 | Self-Supervised Generalisation with Meta Auxiliary Learning (NeurIPS 2019 · 93 citations) |
| 2020 | 96.55 | Self-supervised Label Augmentation via Input Transformations (ICML 2020 · 54 citations) |
What has been tried
102 papers here report 1,173 measured changes (a component added, removed or swapped, compared with a baseline): helped 69%, no clear effect 7%, hurt 20%, mixed 4%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| replace SLR adapters with LoRA | EuroSAT-SAR · accuracy | 85.86 ± 0.3 -> 87.14 ± 0.1 | Hurt | Parameter Efficient Self-Supervised Geospatial Domain Adaptation (CVPR 2024 · 20 citations) |
| Add Radar MIMO Mask augmentation | Radatron test · mAP | Helped | Bootstrapping Autonomous Driving Radars with Self-Supervised Learning (CVPR 2024 · 17 citations) | |
| Add the multi-view contrastive module | BDGP, missing rates 0.9, 0.7, 0.5, 0.3, 0.1 · ACC (%) | 0.5256 -> 0.6032; 0.7016 -> 0.8036; 0.8272 -> 0.9156; 0.8912 -> 0.9304; 0.9404 -> 0.9616 | Helped | Partial Multi-View Clustering via Self-Supervised Network (AAAI 2024 · 11 citations) |
| Add information compression (LIC) | Caltech-2V, IAPR, ESP-Game, MIRFlickr, NUS-Wide · ACC and NMI | Caltech-2V: 63.4 -> 65.9 (ACC), 51.5 -> 52.9 (NMI); IAPR: 47.9 -> 47.7 (ACC), 24.8 -> 24.9 (NMI); ESP-Game: 59.5 -> 59.2 | Mixed | Multi-aspect Self-guided Deep Information Bottleneck for Multi-modal Clustering (AAAI 2025 · 9 citations) |
| use quantile-based negative log-likelihood with q = 0.05 instead of mean-based negative log-likelihood | CIFAR-10 inlier, six outlier datasets · FPR95 and AUROC | Helped | Quantile-Based Maximum Likelihood Training for Outlier Detection (AAAI 2024 · 7 citations) | |
| train the subspace by minimizing mechanical energy instead of using geometric L2 supervision | Square sheet test set · average elastic energy error | 2074 -> 386 | Helped | Neural Modes: Self-supervised Learning of Nonlinear Modal Subspaces (CVPR 2024 · 7 citations) |
| Add structure and surface features | ProteinGym (217 substitution assays) · Spearman | 0.414 -> 0.470 | Helped | Multi-Scale Representation Learning for Protein Fitness Prediction (NeurIPS 2024 · 5 citations) |
| Add ADP pre-training objective | LBA · performance | Helped | Protein-ligand binding representation learning from fine-grained interactions (ICLR 2024 · 4 citations) | |
| increase training-time sample count β | JSP benchmarks · PG | Helped | Self-Labeling the Job Shop Scheduling Problem (NeurIPS 2024 · 4 citations) | |
| Add the ST-Encoder to the static context embedding model | spatial-temporal forecasting datasets · predictive performance | Helped | ST-ReP: Learning Predictive Representations Efficiently for Spatial-Temporal Forecasting (AAAI 2025 · 3 citations) | |
| use FRAD instead of Uni-Mol as molecular encoder | Pocket matching, Kahraman (w/o PO4) · AUC | 0.7870 -> 0.7401 | Hurt | Self-supervised Pocket Pretraining via Protein Fragment-Surroundings Alignment (ICLR 2024 · 3 citations) |
| Do not apply PUS during MBR pre-training | Malicious user detection · AUC | 0.8384 -> 0.8600 | Helped | Pre-trained Behavioral Model for Malicious User Prediction on Social Platform (AAAI 2025 · 2 citations) |
What reviewers will expect
Datasets and baselines named in the 21 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- PCQM4Mv2 · 2 of 21 papers
- ESOL · 2 of 21 papers
- FREESOLV · 2 of 21 papers
- BACE · 2 of 21 papers
- BBBP · 2 of 21 papers
- CLINTOX · 2 of 21 papers
- HIV · 2 of 21 papers
- SIDER · 2 of 21 papers
Baselines
- TS2Vec · 2 of 21 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 64 papers; from OpenAlex and the papers' affiliations).
- Tsinghua University · 6 papers
- Korea Advanced Institute of Science and Technology · 3 papers
- University of Chinese Academy of Sciences · 3 papers
- Apple · 3 papers
- Xi’an Jiaotong University · 2 papers
- Xidian University · 2 papers
- Chinese Academy of Sciences · 2 papers
- ShanghaiTech University · 2 papers
- Adobe · 2 papers
- University of Washington · 2 papers
Use it
Download BibTeX of 40 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Pretext-task visual pre-training (136 papers)
- Self-supervised neuroimaging representations (98 papers)
- Self-supervised speech and brain decoding (56 papers)
Elsewhere in the topic
- Invariance-based self-supervised learning (168 papers)
- Data-efficient and robust self-supervision (122 papers)
- Contrastive versus non-contrastive theory (74 papers)
- Inductive bias and minimal representations (66 papers)
- Masked autoencoder pretraining (48 papers)
- Empirical contrastive visual learning (38 papers)
- Representation evaluation and trade-offs (37 papers)
- Self-supervised speech representations (24 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).