AI Paper Atlas

Research map / Label-efficient and robust learning / Self-supervised and contrastive

Understanding Contrastive Learning Requires Incorporating Inductive Biases

Nikunj Saunshi, Jordan Ash, Surbhi Goel, Dipendra Misra, Cyril Zhang, Sanjeev Arora, Sham Kakade, Akshay Krishnamurthy

ICML 2022 · Label-efficient and robust learning › Self-supervised and contrastive · approach: Inductive bias and minimal representations · 15 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
Use a linear representation instead of an MLPContrastive learning on the Boolean hypercube · Boolean hypercube exampleaccuracy: 74.1 ± 4.3 -> 99.5 ± 0.1Improved
Add weight decay (ablation)MLP + Adam contrastive training · Boolean hypercube exampleaccuracy: 74.1 ± 4.3 -> 89.5 ± 4.9Improved
Train an unconstrained representation that can encode input identityContrastive learning · Boolean hypercube exampleaccuracy: 100 -> 50Hurt
Remove random cropping (ablation)CIFAR-10 SimCLR augmentations · CIFAR-10downstream accuracyMixed
Remove random cropping and horizontal flipping (ablation)CIFAR-10 SimCLR augmentations · CIFAR-10downstream accuracyMixed
Add hash pixels to images (ablation)CIFAR-10 contrastive learning · CIFAR-10downstream accuracyMixed
Train with label-orthogonal modification (ablation)ResNet-18 on CIFAR-10 · CIFAR-10downstream accuracyHurt
Drop 30% of tokens while preserving the order of remaining tokens (ablation)Transformer contrastive learning · AG Newsdownstream accuracyHurt
Randomly permute remaining tokens after dropping 30% (ablation)Transformer contrastive learning · AG Newsdownstream accuracyImproved
Use either the left or right half of each text as an augmentation (ablation)GRU and Transformer contrastive learning · AG Newsdownstream accuracyHurt
Include the original full text as an augmentation (ablation)GRU and Transformer contrastive learning · AG Newsdownstream accuracyImproved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy