Research map / Label-efficient and robust learning / Self-supervised and contrastive
Self-Supervised Contrastive Learning is Approximately Supervised Contrastive Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that contrastive learning (DCL) approximately equals a negatives-only supervised contrastive loss (NSCL) with a gap vanishing as the number of classes grows, characterizes NSCL minimizers (augmentation/within-class collapse, simplex ETF), and bounds few-shot linear-probe error via directional CDNV.
Problem
Theoretical understanding of why label-agnostic self-supervised contrastive learning produces class-clustered representations that transfer to few-shot downstream classification.
Approach
Relates the decoupled contrastive loss to a supervised variant excluding same-class negatives, showing the gap is bounded by log(1+n_max e^2/(N-n_max)) independent of labels and architecture; uses the unconstrained features model to characterize NSCL minimizers and introduces a directional CDNV-based few-shot error bound.</12>
Key results
- Empirically, the DCL-NSCL loss gap is bounded by log(1+e^2/(C-1)) at all epochs and shrinks as the number of classes C grows.
- 100-shot NCCC on CIFAR10: DCL 85.3% vs NSCL 95.7%; LP: DCL 86.3% vs NSCL 95.6%.
- DCL and NSCL-trained models show high representation similarity (RSA/CKA 0.81-0.91).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train with NSCL instead of DCL | SimCLR ResNet-50 · CIFAR10 test, 100-shot | NCCC accuracy: 85.3 $\pm$ $\expnum{2}{1}$ -> 95.7 $\pm$ $\expnum{4}{3}$ | Helped |
| Train with NSCL instead of DCL | SimCLR ResNet-50 · CIFAR10 test, 100-shot | LP accuracy: 86.3 $\pm$ $\expnum{8}{3}$ -> 95.6 $\pm$ $\expnum{4}{3}$ | Helped |
| Train with NSCL instead of DCL | SimCLR ResNet-50 · CIFAR100 test, 100-shot | NCCC accuracy: 57.3 $\pm$ $\expnum{1}{1}$ -> 70.8 $\pm$ $\expnum{1}{1}$ | Helped |
| Train with NSCL instead of DCL | SimCLR ResNet-50 · CIFAR100 test, 100-shot | LP accuracy: 61.7 $\pm$ $\expnum{5}{2}$ -> 73.7 $\pm$ $\expnum{2}{2}$ | Helped |
| Train with NSCL instead of DCL | SimCLR ResNet-50 · mini-ImageNet test, 100-shot | NCCC accuracy: 69.0 $\pm$ $\expnum{2}{1}$ -> 79.8 $\pm$ $\expnum{1}{1}$ | Helped |
| Train with NSCL instead of DCL | SimCLR ResNet-50 · mini-ImageNet test, 100-shot | LP accuracy: 72.9 $\pm$ $\expnum{2}{2}$ -> 81.3 $\pm$ $\expnum{2}{2}$ | Helped |
| Minimize DCL rather than NSCL | SimCLR models · CIFAR10, CIFAR100, mini-ImageNet | few-shot NCCC and LP accuracy | Hurt |
| Train with DCL | ViT-Base · CIFAR10, CIFAR100, mini-ImageNet | same-sample augmentation cosine similarity: 0.92; 0.91; 0.89 | Helped |
| Train DCL and NSCL with matched initialization, minibatch order, and hyperparameters | SimCLR models · CIFAR10, CIFAR100, mini-ImageNet | RSA and CKA representation similarity: ResNet-50: 0.83, 0.81; 0.91, 0.91; 0.87, 0.90. ViT-Base: 0.86, 0.82; 0.91, 0.91; 0.89, 0.89 | Helped |
Surprising findings
- DCL training reduces directional CDNV by about an order of magnitude while only modestly lowering CDNV.
Limitations
- Bound in the few-shot proposition has large constants and scales linearly with C', limiting practical utility.
- Closeness of DCL and NSCL loss values does not imply closeness of minimizers or representations in parameter space.
- Global minimizer characterization relies on the unconstrained features model.
Experimental setup
Datasets: CIFAR10 CIFAR100 mini-ImageNet Tiny-ImageNet SVHN ImageNet-1K
Benchmarks: CIFAR10 CIFAR100 mini-ImageNet
Metrics: NCCC accuracy linear probing accuracy DCL and NSCL loss gap CDNV directional CDNV RSA CKA
Baselines: MoCo v2 NSCL
Builds on: SimCLR decoupled contrastive learning neural collapse supervised contrastive loss class-distance-normalized variance
Closest papers by meaning
- On the Alignment Between Supervised and Self-Supervised Contrastive Learning (ICLR 2026 · similarity 0.93)
- Self-Contrastive Learning: Single-Viewed Supervised Contrastive Framework Using Sub-network (AAAI 2023 · similarity 0.92)
- Supervised Contrastive Learning (NeurIPS 2020 · similarity 0.91)
- On the duality between contrastive and non-contrastive self-supervised learning (ICLR 2023 · similarity 0.90)
- An Augmentation-Aware Theory for Self-Supervised Contrastive Learning (ICML 2025 · similarity 0.90)
- On the Similarities of Embeddings in Contrastive Learning (ICML 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.