Research map / Vision-language and multimodal / Cross-modal learning
On the Comparison between Multi-modal and Single-modal Contrastive Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Theoretical feature-learning analysis of single-modal vs multi-modal InfoMax contrastive learning on a signal-noise data model with a ReLU network, showing multi-modal training generalizes better downstream because the high-SNR second modality boosts signal learning.
Problem
Lack of a unified theoretical understanding of why multi-modal contrastive learning (e.g., CLIP) outperforms single-modal contrastive learning in downstream out-of-distribution generalization, covering both optimization and generalization.
Approach
Models each modality as signal plus noise, trains ReLU encoders with an InfoMax-style contrastive loss under gradient descent, and tracks signal learning versus noise memorization via a two-stage trajectory analysis; the cross-modal cooperation through contrastive learning raises signal learning, so multi-modal training generalizes better than single-modal training.
Key results
- On ColoredMNIST, multi-modal contrastive learning reaches 82.13% OOD test accuracy versus 12.68% for single-modal learning.
- In synthetic experiments, multi-modal OOD test accuracy converges near 1.0 while single-modal stagnates around 0.5.
- Theory shows single-modal contrastive learning reaches training error at most epsilon but test error Theta(1), whereas multi-modal achieves o(1) test error.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use multi-modal rather than single-modal contrastive learning | Contrastive learning on ColoredMNIST · ColoredMNIST test | accuracy: 12.68% -> 82.13% | Helped |
| use multi-modal rather than single-modal contrastive learning | Contrastive learning on ColoredMNIST · ColoredMNIST train | accuracy: 88.43% -> 87.77% | Mixed |
| use multi-modal rather than single-modal contrastive learning | Contrastive learning on synthetic data · Synthetic OOD test | accuracy | Helped |
| use multi-modal rather than single-modal contrastive learning | Contrastive learning on synthetic data · Synthetic OOD test | noise memorization | Helped |
| rely on data augmentation alone rather than a second modality | Single-modal contrastive learning · Downstream tasks | generalization | No clear effect |
Surprising findings
- Single-modal contrastive learning achieves near-zero training error yet has Theta(1) downstream test error, i.e. it memorizes noise.
- Data augmentation alone cannot improve single-modal contrastive learning because augmentation preserves the original SNR.
Limitations
- The data model is simplified (single-layer ReLU, linear signal and Gaussian noise) and may not reflect practical multi-modal data.
- Analysis requires many assumptions such as high dimensionality, specific sample-size/SNR relations, and matched dimensions and noise levels across modalities.
- Real-world experiment uses an idealized language encoder that maps captions to one-hot labels, and a small single-run comparison.
Experimental setup
Datasets: synthetic signal-noise data ColoredMNIST
Benchmarks: ColoredMNIST (OOD color-shift test split) synthetic OOD test data
Metrics: test accuracy training loss train accuracy
Baselines: Single-modal contrastive learning
Builds on: InfoMax contrastive loss SimCLR CLIP
Closest papers by meaning
- Multi-modal contrastive learning adapts to intrinsic dimensions of shared latent variables (NeurIPS 2025 · similarity 0.90)
- Connect, Collapse, Corrupt: Learning Cross-Modal Tasks with Uni-Modal Data (ICLR 2024 · similarity 0.88)
- Understanding the Robustness of Multi-modal Contrastive Learning to Distribution Shift (ICLR 2024 · similarity 0.88)
- Quantifying and Enhancing Multi-modal Robustness with Modality Preference (ICLR 2024 · similarity 0.87)
- The "Law'' of the Unconscious Contrastive Learner: Probabilistic Alignment of Unpaired Modalities (ICLR 2025 · similarity 0.87)
- Continual Multimodal Contrastive Learning (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.