Research map / Label-efficient and robust learning / Recognition and long-tail
Bridging Mini-Batch and Asymptotic Analysis in Contrastive Learning: From InfoNCE to Kernel-Based Losses
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that InfoNCE-family contrastive losses share hyperspherical-energy minimisers in the mini-batch and asymptotic regimes, and proposes DHEL and kernel contrastive learning (KCL) losses whose expectations are batch-size independent, evaluated with ResNet18/50 on CIFAR10/100, STL-10 and ImageNet-100.
Problem
Contrastive learning losses such as InfoNCE, SimCLR and DCL are poorly understood theoretically: their optima differ in mini-batch and asymptotic regimes, their expected losses depend on batch size, and they suffer from temperature sensitivity, small-batch reliance and dimensionality collapse.
Approach
Proves that InfoNCE-type losses share the regular-simplex mini-batch optimum and the uniform-on-sphere asymptotic optimum, introduces DHEL, which decouples the uniformity denominator from positives, and KCL, a kernel-based two-term loss whose expectation is independent of batch size so its minimisers can be identified non-asymptotically.
Key results
- DHEL significantly outperforms SimCLR across all datasets and batch sizes, with its median accuracy always higher than DCL's.
- Kernel methods outperform both SimCLR and DCL, and in several cases improve further upon DHEL.
- DHEL utilises more dimensions; in CIFAR100 it uses more than double the dimensions of competitors (max 128).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace the InfoNCE denominator with a negative-only denominator (DHEL) | Contrastive learning pretraining · CIFAR10, CIFAR100, STL-10, ImageNet-100 | median linear-evaluation accuracy | Helped |
| Use Gaussian or logarithmic kernel contrastive loss instead of SimCLR or DCL | Contrastive learning pretraining · CIFAR10, CIFAR100, STL-10, ImageNet-100 | median linear-evaluation accuracy | Helped |
| Use DHEL or kernel contrastive loss at small batch sizes | Contrastive learning pretraining · CIFAR10, CIFAR100, STL-10, ImageNet-100 | median linear-evaluation accuracy | Helped |
| Use DHEL or kernel contrastive loss across tested hyperparameter values | Contrastive learning pretraining · CIFAR10, CIFAR100, STL-10, ImageNet-100 | accuracy variation across hyperparameters | Helped |
| Use DHEL rather than competing methods | Contrastive learning representations · CIFAR100 | rank | Helped |
| Use Gaussian KCL | Contrastive learning representations · CIFAR10, CIFAR100, STL-10, ImageNet-100 | rank | Helped |
| Use DHEL rather than the compared methods | Contrastive learning representations · CIFAR10, CIFAR100 | uniformity | Helped |
| Use SimCLR or DCL rather than DHEL | Contrastive learning representations · CIFAR10, CIFAR100 | alignment | Helped |
| Increase temperature | Contrastive learning methods · CIFAR10, CIFAR100 | linear-evaluation accuracy | Hurt |
| Vary temperature | Gaussian KCL · CIFAR10, CIFAR100 | linear-evaluation accuracy | No clear effect |
Surprising findings
- The expected InfoNCE, SimCLR and DCL losses depend on batch size even after normalisation, so their asymptotic analysis is needed for optima.
Limitations
- Analysis of InfoNCE variants does not address the non-asymptotic optima that arise in practice.
- Experiments are limited to small/medium image datasets and the authors note large-scale evaluation is needed.
- The reason DHEL favours uniformity over alignment is not clearly explained.
Experimental setup
Datasets: CIFAR10 CIFAR100 STL-10 ImageNet-100
Benchmarks: CIFAR10 CIFAR100 STL-10 ImageNet-100
Metrics: linear evaluation accuracy alignment uniformity rank effective rank Wasserstein distance between similarity distributions
Baselines: SimCLR DCL
Builds on: InfoNCE SimCLR DCL HSIC Neural collapse
Closest papers by meaning
- Why do We Need Large Batchsizes in Contrastive Learning? A Gradient-Bias Perspective (NeurIPS 2022 · similarity 0.90)
- On the Similarities of Embeddings in Contrastive Learning (ICML 2025 · similarity 0.88)
- Self-Supervised Contrastive Learning is Approximately Supervised Contrastive Learning (NeurIPS 2025 · similarity 0.87)
- BECLR: Batch Enhanced Contrastive Few-Shot Learning (ICLR 2024 · similarity 0.86)
- Engineering the Neural Collapse Geometry of Supervised-Contrastive Loss (Student Abstract) (AAAI 2024 · similarity 0.86)
- Your contrastive learning problem is secretly a distribution alignment problem (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.