Research map / Label-efficient and robust learning / Self-supervised and contrastive
Understanding Contrastive Learning Requires Incorporating Inductive Biases
Nikunj Saunshi, Jordan Ash, Surbhi Goel, Dipendra Misra, Cyril Zhang, Sanjeev Arora, Sham Kakade, Akshay Krishnamurthy
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use a linear representation instead of an MLP | Contrastive learning on the Boolean hypercube · Boolean hypercube example | accuracy: 74.1 ± 4.3 -> 99.5 ± 0.1 | Improved |
| Add weight decay (ablation) | MLP + Adam contrastive training · Boolean hypercube example | accuracy: 74.1 ± 4.3 -> 89.5 ± 4.9 | Improved |
| Train an unconstrained representation that can encode input identity | Contrastive learning · Boolean hypercube example | accuracy: 100 -> 50 | Hurt |
| Remove random cropping (ablation) | CIFAR-10 SimCLR augmentations · CIFAR-10 | downstream accuracy | Mixed |
| Remove random cropping and horizontal flipping (ablation) | CIFAR-10 SimCLR augmentations · CIFAR-10 | downstream accuracy | Mixed |
| Add hash pixels to images (ablation) | CIFAR-10 contrastive learning · CIFAR-10 | downstream accuracy | Mixed |
| Train with label-orthogonal modification (ablation) | ResNet-18 on CIFAR-10 · CIFAR-10 | downstream accuracy | Hurt |
| Drop 30% of tokens while preserving the order of remaining tokens (ablation) | Transformer contrastive learning · AG News | downstream accuracy | Hurt |
| Randomly permute remaining tokens after dropping 30% (ablation) | Transformer contrastive learning · AG News | downstream accuracy | Improved |
| Use either the left or right half of each text as an augmentation (ablation) | GRU and Transformer contrastive learning · AG News | downstream accuracy | Hurt |
| Include the original full text as an augmentation (ablation) | GRU and Transformer contrastive learning · AG News | downstream accuracy | Improved |
Closest papers by meaning
- The Power of Contrast for Feature Learning: A Theoretical Analysis (JMLR 2023 · similarity 0.89)
- A theoretical study of inductive biases in contrastive learning (ICLR 2023 · similarity 0.88)
- Contrastive Learning Inverts the Data Generating Process (ICML 2021 · similarity 0.88)
- A Statistical Theory of Contrastive Learning via Approximate Sufficient Statistics (NeurIPS 2025 · similarity 0.87)
- An Augmentation-Aware Theory for Self-Supervised Contrastive Learning (ICML 2025 · similarity 0.87)
- Self-Supervised Contrastive Learning is Approximately Supervised Contrastive Learning (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
