Research map / Label-efficient and robust learning / Self-supervised and contrastive
Inductive bias and minimal representations
Examines inductive bias and information content of contrastive representations and proposes variants.
66 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2020–2026) · an approach in Self-supervised and contrastive, keyword cluster “supervised contrastive · contrastive self · negative”
State History Tried Checklist Preprints Who Export
Where it stands
Not enough benchmark data Fewer than four papers report the same benchmark, so there is no record to follow.
18 papers in 2023–24 and 21 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic grew, from 5.6% to 6.5%.
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2020 · What Makes for Good Views for Contrastive Learning? (NeurIPS 2020 · 640 citations)
- 2021 · AdCo: Adversarial Contrast for Efficient Learning of Unsupervised Representations From Self-Trained Negative Adversaries (CVPR 2021 · 117 citations)
- 2022 · Use All the Labels: A Hierarchical Multi-Label Contrastive Learning Framework (CVPR 2022 · 86 citations)
- 2023 · Best of Both Worlds: Multimodal Contrastive Learning With Tabular and Imaging Data (CVPR 2023 · 70 citations)
- 2024 · Your contrastive learning problem is secretly a distribution alignment problem (NeurIPS 2024 · 7 citations)
- 2025 · Enhancing Contrastive Learning Inspired by the Philosophy of “The Blind Men and the Elephant” (AAAI 2025 · 4 citations)
- 2026 · The Loss Is Not Enough: Sampling Conditions and Inductive Bias in Contrastive Representation Learning (ICML 2026 · 0 citations)
Earliest papers here
- What Makes for Good Views for Contrastive Learning? (NeurIPS 2020 · 640 citations)
- Hard Negative Mixing for Contrastive Learning (NeurIPS 2020 · 264 citations)
- Debiased Contrastive Learning (NeurIPS 2020 · 17 citations)
What has been tried
51 papers here report 576 measured changes (a component added, removed or swapped, compared with a baseline): helped 67%, no clear effect 7%, hurt 22%, mixed 4%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| Increase the domain alignment weight | PACS domain classification · accuracy | 72.11% to 95.16% | Helped | Your contrastive learning problem is secretly a distribution alignment problem (NeurIPS 2024 · 7 citations) |
| Set JointCrop β to 2 instead of 0 | Tiny-ImageNet · top-1 accuracy | 47.53 -> 46.69 | No clear effect | Enhancing Contrastive Learning Inspired by the Philosophy of “The Blind Men and the Elephant” (AAAI 2025 · 4 citations) |
| Apply truncated SVD at a selected q | CIFAR-10, T1 · top-1 accuracy (%) | 68.82 -> 69.87 | Helped | How does Labeling Error Impact Contrastive Learning? A Perspective from Data Dimensionality Reduction (ICML 2025 · 0 citations) |
| Use alpha 1 instead of alpha 2 for the bottleneck regularizer | CIFAR-100 · linear probing accuracy | 59.38 -> 59.06 | Hurt | Projection Head is Secretly an Information Bottleneck (ICLR 2025 · 0 citations) |
| Set the perturbation factor to δ=0.25 rather than δ=0.01, 0.05, 0.10, or 0.50 | ESC-50 · score | 47.55; 47.90; 48.40; 58.90; 41.85 | Helped | Contrastive Learning from Synthetic Audio Doppelgängers (ICLR 2025 · 0 citations) |
| Replace the Sentence Transformer graph with a WordNet hierarchy-distance graph | ImageNet-9 Same Class · accuracy | 66.5 -> 55.7 | Hurt | $\mathbb{X}$-Sample Contrastive Loss: Improving Contrastive Learning with Sample Similarity Graphs (ICLR 2025 · 0 citations) |
| Add SAQE (query-expanded self-attention) | PEMS, GZ, CITY, METR, SG, BEAUTY, MOVIE&TV, GOWALLA · MAE | 3.14%-19.14% | Helped | High-Order Contrastive Learning with Fine-grained Comparative Levels for Sparse Ordinal Tensor Completion (ICML 2024) |
What reviewers will expect
Datasets and baselines named in the 14 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- CIFAR-100 · 4 of 14 papers
- CIFAR-10 · 3 of 14 papers
- ImageNet-1K · 2 of 14 papers
- STL-10 · 2 of 14 papers
- Tiny-ImageNet · 2 of 14 papers
- ImageNet-100 · 2 of 14 papers
Baselines
- SimCLR · 3 of 14 papers
- Barlow Twins · 2 of 14 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 39 papers; from OpenAlex and the papers' affiliations).
- Peking University · 2 papers
- Korea Advanced Institute of Science and Technology · 2 papers
- Tsinghua University · 2 papers
- Samsung · 2 papers
- University of Electronic Science and Technology of China · 2 papers
Use it
Download BibTeX of 40 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Contrastive versus non-contrastive theory (74 papers)
- Empirical contrastive visual learning (38 papers)
- Representation evaluation and trade-offs (37 papers)
Elsewhere in the topic
- Invariance-based self-supervised learning (168 papers)
- Pretext-task visual pre-training (136 papers)
- Data-efficient and robust self-supervision (122 papers)
- Self-supervision in supervised pipelines (121 papers)
- Self-supervised neuroimaging representations (98 papers)
- Self-supervised speech and brain decoding (56 papers)
- Masked autoencoder pretraining (48 papers)
- Self-supervised speech representations (24 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).