Research map / Label-efficient and robust learning / Recognition and long-tail
Contrasting Multiple Representations with the Multi-Marginal Matching Gap
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes the multi-marginal matching gap (M3G), a contrastive loss for k≥3 views that contrasts ground-truth k-tuple matching cost with the multi-marginal OT polymatching cost, solved via a multi-marginal Sinkhorn algorithm.
Problem
Contrastive/joint-embedding learning with k≥3 views or modalities is usually done by extending pairwise losses (averaging over pairs or one-vs-average-of-rest), which ignore the joint coherence of k-tuples.
Approach
Define a loss as the optimality gap between the ground-truth k-tuple matching and the best multi-marginal OT polymatching under a circular-variance multiway cost, using entropic multi-marginal Sinkhorn; the gradient follows from Fenchel-Young theory and needs only the optimal polystochastic tensor, not a backward pass through the solver.
Key results
- On ImageNet-1k with ViT-B/16 (n=64, 300 epochs), M3G reaches 75.61% at k=3 and 75.75% at k=4 linear accuracy, versus 75.36% for InfoNCE_pwe at k=3.
- On DomainNet unseen-domain evaluation, M3G is ranked first in most settings with average rank 1.4 across 30 linear evaluations, and a mean 3.1% improvement over the second-best model.
- On EEG sleep staging with k=6 channels, M3G (eps=0.2) beats InfoNCE_pwe at s=100 (56.1 vs 52.2) and s=1000 (64.6 vs 56.3).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use M3G instead of the strongest pairwise baseline, InfoNCE_pwe | ImageNet-1k multiview pretraining, ViT-B/16 · ImageNet-1k, k=3 views | linear top-1 accuracy: 75.36 \pm0.10 -> 75.61 \pm 0.12 | Helped |
| Use M3G instead of the strongest pairwise baseline, InfoNCE_pwe | ImageNet-1k multiview pretraining, ViT-B/16 · ImageNet-1k, k=4 views | linear top-1 accuracy: 75.26 \pm 0.16 -> 75.75 \pm 0.11 | Helped |
| Use M3G instead of the strongest listed pairwise baseline | DomainNet cross-domain representation learning · DomainNet, unseen clp | prediction accuracy: 24.1 \pm 0.2 -> 32.4 \pm 0.3 | Helped |
| Use M3G instead of the strongest listed pairwise baseline | DomainNet cross-domain representation learning · DomainNet, unseen inf | prediction accuracy: 11.0 \pm 0.2 -> 12.2 \pm 0.1 | Helped |
| Use M3G instead of the strongest listed pairwise baseline | DomainNet cross-domain representation learning · DomainNet, unseen pnt | prediction accuracy: 30.6 \pm 0.6 -> 31.3 \pm 0.1 | Helped |
| Use M3G instead of the strongest listed pairwise baseline | DomainNet cross-domain representation learning · DomainNet, unseen qdr | prediction accuracy: 8.8 \pm 0.2 -> 10.4 \pm 0.3 | Helped |
| Use M3G instead of the strongest listed pairwise baseline | DomainNet cross-domain representation learning · DomainNet, unseen rel | prediction accuracy: 44.6 \pm 0.1 -> 46.3 \pm 0.2 | Helped |
| Use M3G instead of the strongest listed pairwise baseline | DomainNet cross-domain representation learning · DomainNet, unseen skt | prediction accuracy: 24.9 \pm 0.1 -> 26.5 \pm 0.4 | Helped |
| Use M3G instead of pairwise InfoNCE | PhysioNet EEG channel representation learning · PhysioNet EEG, 10 samples per class | prediction accuracy: 35.5 -> 36.6 | Helped |
| Use M3G instead of pairwise InfoNCE | PhysioNet EEG channel representation learning · PhysioNet EEG, 50 samples per class | prediction accuracy: 49.3 -> 49.2 | Hurt |
| Use M3G instead of pairwise InfoNCE | PhysioNet EEG channel representation learning · PhysioNet EEG, 100 samples per class | prediction accuracy: 52.2 -> 56.1 | Helped |
| Use M3G instead of pairwise InfoNCE | PhysioNet EEG channel representation learning · PhysioNet EEG, 1000 samples per class | prediction accuracy: 56.3 -> 64.6 | Helped |
Surprising findings
- For k=2, M3G does not reduce to InfoNCE and gives 74.75% vs 74.61% for InfoNCE, a near-tie.
- In DomainNet, the one-vs-average-of-rest baseline outperforms the pairwise variant, reversing the ImageNet trend.
Limitations
- Cost scales as n^k in batch size and number of views, limiting use to small batches (n=16-64) and k≤6.
- Entropic regularization epsilon may need to scale with batch size; no principled setting provided.
- Only a generic circular-variance multiway cost was explored; domain-specific costs left to future work.
Experimental setup
Datasets: ImageNet-1k DomainNet PhysioNet Challenge 2018 SleepEDFx
Benchmarks: ImageNet-1k linear evaluation DomainNet unseen-domain classification SleepEDFx sleep staging
Metrics: top-1 linear accuracy prediction accuracy average rank
Baselines: BYOL_pwe BYOL_ave InfoNCE_pwe InfoNCE_ave M3G with csd cost
Builds on: BYOL SimCLR InfoNCE Fenchel-Young losses multi-marginal Sinkhorn
Closest papers by meaning
- Disentangled Representation Learning with the Gromov-Monge Gap (ICLR 2025 · similarity 0.86)
- Joint Metric Space Embedding by Unbalanced Optimal Transport with Gromov–Wasserstein Marginal Penalization (ICML 2025 · similarity 0.85)
- Rethinking Multi-view Representation Learning via Distilled Disentangling (CVPR 2024 · similarity 0.84)
- Deep Incomplete Multi-view Learning via Cyclic Permutation of VAEs (ICLR 2025 · similarity 0.84)
- $\mathbb{X}$-Sample Contrastive Loss: Improving Contrastive Learning with Sample Similarity Graphs (ICLR 2025 · similarity 0.84)
- I-Con: A Unifying Framework for Representation Learning (ICLR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.