Research map / Label-efficient and robust learning / Recognition and long-tail
On the Similarities of Embeddings in Contrastive Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Unifies contrastive losses via cosine similarity statistics, showing full-batch excessive negative separation and that small mini-batches raise negative-pair similarity variance, and proposes a variance-reducing auxiliary loss (VRNS).
Problem
Existing theory on contrastive learning covers only specific losses and full-batch optima; it is unclear how negative-pair similarity behaves in full-batch versus mini-batch training, and why small batches hurt performance.
Approach
Analyze embedding similarities (mean and variance of positive/negative pair cosine similarity) for a broad class of InfoNCE-based and independently additive losses; show full-batch excessive separation of negatives when within-view negatives are omitted, and that mini-batch training inflates negative-similarity variance; add an auxiliary loss pushing negative similarities toward -1/(n-1).
Key results
- Variance of negative-pair similarities for SimCLR rises from 0.1396 at batch 512 to 0.1649 at batch 32; adding the VRNS loss lowers it to 0.1008 at batch 32.
- The proposed auxiliary loss gives consistent accuracy gains for SimCLR, DCL and DHEL on CIFAR, with larger gains at smaller batch sizes.
- Theory: with (c1,c2)=(1,0) and a sufficiently small sigmoid bias b, full-batch optima exhibit excessive separation (positive similarity < 1, negative similarity < -1/(n-1)).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add VRNS auxiliary loss to SimCLR | SimCLR · CIFAR-100 (batch size 32) | variance of negative-pair similarities: 0.1649 -> 0.1008 | Helped |
| Add VRNS auxiliary loss to SimCLR | SimCLR · CIFAR-100 (batch size 64) | variance of negative-pair similarities: 0.1505 -> 0.0952 | Helped |
| Add VRNS auxiliary loss to SimCLR | SimCLR · CIFAR-100 (batch size 128) | variance of negative-pair similarities: 0.1444 -> 0.0929 | Helped |
| Add VRNS auxiliary loss to SimCLR | SimCLR · CIFAR-100 (batch size 256) | variance of negative-pair similarities: 0.1404 -> 0.0921 | Helped |
| Add VRNS auxiliary loss to SimCLR | SimCLR · CIFAR-100 (batch size 512) | variance of negative-pair similarities: 0.1396 -> 0.0917 | Helped |
| Add VRNS auxiliary loss | SimCLR · CIFAR-10 and CIFAR-100 (various temperatures) | classification accuracy | Helped |
| Add VRNS auxiliary loss | SimCLR, DCL, and DHEL · CIFAR datasets (various batch sizes) | top-1 classification accuracy | Helped |
Surprising findings
- Under the sigmoid (SigLIP-style) loss, small bias b can cause excessive separation of negatives even in full-batch training.
Limitations
- The variance reduction may also suppress meaningful intrinsic variance in the data structure.
- Analysis is mainly for the unimodal case and relies on idealized assumptions such as dimension d >= n-1 and normalized embeddings.
- Empirical validation is limited to ResNet-18 on CIFAR and ImageNet with SimCLR-style losses.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 ImageNet
Benchmarks: CIFAR-10 CIFAR-100 ImageNet
Metrics: top-1 classification accuracy variance of negative-pair similarities
Baselines: SimCLR DCL DHEL SigLIP
Builds on: InfoNCE SimCLR sigmoid contrastive loss (SigLIP) decoupled contrastive loss spectral contrastive loss
Closest papers by meaning
- Breaking the Batch Barrier (B3) of Contrastive Learning via Smart Batch Mining (NeurIPS 2025 · similarity 0.90)
- $\mathbb{X}$-Sample Contrastive Loss: Improving Contrastive Learning with Sample Similarity Graphs (ICLR 2025 · similarity 0.90)
- Diversity Is All You Need for Contrastive Learning: Spectral Bounds on Gradient Magnitudes (NeurIPS 2025 · similarity 0.90)
- Self-Supervised Contrastive Learning is Approximately Supervised Contrastive Learning (NeurIPS 2025 · similarity 0.89)
- A Statistical Theory of Contrastive Learning via Approximate Sufficient Statistics (NeurIPS 2025 · similarity 0.88)
- Your contrastive learning problem is secretly a distribution alignment problem (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.