Research map / Label-efficient and robust learning / Self-supervised and contrastive
Multi-modal contrastive learning adapts to intrinsic dimensions of shared latent variables
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Theoretical analysis of multi-modal contrastive (CLIP-style infoNCE) learning with temperature optimization, showing minimizers are sufficient, similarity-maximizing, and adapt to the intrinsic dimension of shared latent variables, validated on synthetic and CITE-seq data.
Problem
Existing theory of CLIP-style multi-modal contrastive learning assumes linear representations, factor-model data, or a fixed/ignored temperature, and cannot explain why learned representations adapt to a lower intrinsic dimension than the user-specified output dimension.
Approach
Using a discretized, fine-grained mutual-information order and a variational decomposition of the infoNCE loss with optimized temperature, the paper shows that infoNCE minimizers over general measurable function classes are aligned, sufficient, have constant positive-pair similarity, temperature tending to zero, and intrinsic dimension equal to the shared-latent dimension k*, without requiring uni
Key results
- On synthetic data with k*=5 and d1=d2=20, estimated intrinsic dimension saturates near the true value 5 once output dimension d exceeds 5 in both linear and nonlinear settings (averaged over 50 repetitions).
- On CITE-seq, the MLE-estimated intrinsic dimension of in-sample representations does not exceed 10 regardless of output dimension, and accuracy saturates above output dimension 10.
- Empirically, the optimized temperature converges toward zero and positive-pair similarities concentrate at a constant that dominates negative pairs.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| increase output dimension beyond 5 | multi-modal contrastive learning with 5-layer ReLU encoders · Synthetic data, linear and nonlinear settings | top-1 matching accuracy | No clear effect |
| increase output dimension beyond 5 | multi-modal contrastive learning with 5-layer ReLU encoders · Synthetic data, linear and nonlinear settings | estimated intrinsic dimension: k^* = 5 | No clear effect |
| train with optimized temperature | multi-modal contrastive learning · Synthetic data, linear and nonlinear settings | temperature | Helped |
| increase output dimension beyond 10 | multi-modal contrastive learning with 5-layer ReLU encoders · CITE-seq healthy human bone marrow cells | matching accuracy | No clear effect |
| increase output dimension beyond 10 | multi-modal contrastive learning with 5-layer ReLU encoders · CITE-seq healthy human bone marrow cells | estimated intrinsic dimension: 10 | No clear effect |
| learn representations with multi-modal contrastive loss | 5-layer ReLU encoders · CITE-seq healthy human bone marrow cells | cell-type classification accuracy | Helped |
Surprising findings
- Learned representations prefer intrinsic dimension 2 rather than the specified output dimension 3 in the motivating linear example, indicating adaptation below the user-specified dimension.
Limitations
- Theory assumes the aligned, mutually informative set V(H) is nonempty and works with population-level loss rather than finite samples.
- Characterization of the precise limit of optimized temperature beyond the nonempty-V regime is left open.
- Real-data experiments use a CITE-seq dataset and intrinsic-dimension estimates from an MLE estimator, not a CLIP image-text model trained at scale.
Experimental setup
Datasets: synthetic Gaussian linear and nonlinear data CITE-seq healthy human bone marrow cells (via Seurat) ImageNetV2 (appendix) YFCC (appendix)
Benchmarks: CITE-seq top-0.5% matching accuracy CITE-seq cell-type classification synthetic top-1 matching accuracy
Metrics: top-alpha% matching accuracy classification accuracy MLE intrinsic dimension estimate temperature convergence similarity histograms
Baselines: linear-representation CLIP theory alignment-uniformity analysis
Builds on: CLIP infoNCE alignment and uniformity (Wang & Isola) infoMax principle
Closest papers by meaning
- On the Comparison between Multi-modal and Single-modal Contrastive Learning (NeurIPS 2024 · similarity 0.90)
- Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learning (NeurIPS 2022 · similarity 0.89)
- Model-Aware Contrastive Learning: Towards Escaping the Dilemmas (ICML 2023 · similarity 0.88)
- Variational Supervised Contrastive Learning (NeurIPS 2025 · similarity 0.87)
- Separating common from salient patterns with Contrastive Representation Learning (ICLR 2024 · similarity 0.87)
- The "Law'' of the Unconscious Contrastive Learner: Probabilistic Alignment of Unpaired Modalities (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.