Research map / Label-efficient and robust learning / Self-supervised and contrastive

Your contrastive learning problem is secretly a distribution alignment problem

NeurIPS 2024 · Label-efficient and robust learning › Self-supervised and contrastive · approach: Inductive bias and minimal representations · 7 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Reframes contrastive learning losses (InfoNCE, RINCE, BYOL) as proximal/Bregman-projection alignment problems with entropic optimal transport, yielding multistep GCA variants and unbalanced OT losses (GCA-UOT).

Problem

Contrastive learning's theoretical foundations and the mechanisms by which it builds representations are poorly understood; existing losses lack flexible control over alignment constraints and are fragile to noisy views.

Approach

Interpret InfoNCE, RINCE and BYOL as single-step Bregman/proximal projections of the augmentation kernel onto marginal constraint sets; iterating Sinkhorn-style projections (and relaxing marginals with unbalanced OT) gives a family of GCA losses with a customizable target transport plan.

Key results

• GCA-UOT achieves the top accuracy on all four standard datasets in the reported table (e.g., 93.50% on CIFAR-10 with ResNet-18 vs 93.27% for best INCE-based baseline column shown).
• On PACS, increasing domain alignment weight raises domain classification accuracy from 72.11% to 95.16% without hurting class accuracy.
• GCA-INCE adds about 5% FLOPs over INCE (6.65 vs 6.31 MFlops), while GCA-UOT saves about 30% (4.54 MFlops).

Measured effects reported in the paper

Limitations

• Theoretical equivalences rely on idealized assumptions (expressivity of encoder, small intra-class variance) for the uniformity-to-CE claim.
• Gains over baselines are often small (fractions of a percentage point) relative to reported standard deviations.

Experimental setup

Datasets: CIFAR-10 CIFAR-100 SVHN ImageNet100 CIFAR-10C PACS

Benchmarks: CIFAR-10 CIFAR-100 SVHN ImageNet100 CIFAR-10C PACS

Metrics: test accuracy (%)

Baselines: INCE RINCE BYOL SimCLR IOT IOT-uni

Builds on: InfoNCE RINCE BYOL SimCLR Sinkhorn algorithm

Closest papers by meaning

• Understanding and Generalizing Contrastive Learning from the Inverse Optimal Transport Perspective (ICML 2023 · similarity 0.90)
• The Loss Is Not Enough: Sampling Conditions and Inductive Bias in Contrastive Representation Learning (ICML 2026 · similarity 0.89)
• On the Similarities of Embeddings in Contrastive Learning (ICML 2025 · similarity 0.88)
• $\mathbb{X}$-Sample Contrastive Loss: Improving Contrastive Learning with Sample Similarity Graphs (ICLR 2025 · similarity 0.87)
• Aligning Visual Contrastive learning models via Preference Optimization (ICLR 2025 · similarity 0.87)
• BECLR: Batch Enhanced Contrastive Few-Shot Learning (ICLR 2024 · similarity 0.87)