Research map / Vision-language and multimodal / Cross-modal learning
Contrasting with Symile: Simple Model-Agnostic Representation Learning for Unlimited Modalities
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Symile, a contrastive objective built on a multi-sample total correlation lower bound and a multilinear inner product, that captures higher-order information across any number of modalities and beats pairwise CLIP on XOR, a multilingual 3-modality dataset, and clinical CXR/ECG/labs retrieval.
Problem
Pairwise CLIP-style contrastive objectives capture only pairwise dependencies between modalities and miss higher-order (conditional) information needed when many modalities must be integrated jointly.
Approach
Replace the pairwise dot-product critic with a multilinear inner product (coordinate-wise sum of elementwise products) and derive a contrastive lower bound on total correlation, so the learned modality-specific representations capture all pairwise and higher-order information and form sufficient statistics for predicting the remaining modalities.
Key results
- On SYMILE-M3-2/5/10, Symile reaches 0.939/0.919/0.882 retrieval accuracy versus CLIP's 0.473/0.187/0.094.
- With 12.5% complete training triples on SYMILE-M3-2 (missing-data setting), Symile achieves 0.906 accuracy, far above CLIP's 0.473.
- On clinical CXR retrieval from ECG and labs (10 candidates), Symile gets 0.435 vs CLIP's 0.387 accuracy.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Symile instead of pairwise CLIP | Multimodal retrieval on Symile-M3-2 · Symile-M3-2 | accuracy: 0.473 -> 0.939 | Helped |
| Use Symile instead of pairwise CLIP | Multimodal retrieval on Symile-M3-5 · Symile-M3-5 | accuracy: 0.187 -> 0.919 | Helped |
| Use Symile instead of pairwise CLIP | Multimodal retrieval on Symile-M3-10 · Symile-M3-10 | accuracy: 0.094 -> 0.882 | Helped |
| Train Symile with modalities independently missing with probability 0.5 | Symile on Symile-M3-2 · Symile-M3-2, 12.5% complete training data | accuracy: 0.473 -> 0.906 ± 3.4 × 10^{-4} | Helped |
| Use Symile instead of pairwise CLIP | Chest X-ray retrieval from ECG and labs · Symile-MIMIC | accuracy: 0.387 ± 0.003 -> 0.435 ± 0.007 | Helped |
| Increase p̂ from 0 toward 1, adding higher-order dependence | Symile synthetic representation model · Synthetic data | accuracy: null | Helped |
| Increase p̂ when predicting the third modality from the other two | Pairwise CLIP synthetic representation model · Synthetic data | accuracy: null | No clear effect |
| Use pairwise CLIP for the three-modality XOR prediction task | Three-modality XOR model · Three-modality XOR | accuracy: 0.5 | Hurt |
Surprising findings
- On the XOR setting, pairwise CLIP performs no better than random chance (0.5 accuracy) even though the variables are deterministically related.
Limitations
- May be less sample efficient than CLIP when only pairwise statistics matter.
- Negative sampling approximation's scaling beyond three modalities is untested.
- Zero-shot predictions are not Bayes optimal when the marginal of the target is non-uniform.
Experimental setup
Datasets: Common Voice ImageNet MIMIC-IV MIMIC-CXR SYMILE-M3 (constructed) SYMILE-MIMIC (constructed)
Benchmarks: SYMILE-M3 zero-shot retrieval SYMILE-MIMIC CXR retrieval synthetic XOR retrieval
Metrics: accuracy mean accuracy over bootstrap samples
Baselines: CLIP (pairwise)
Builds on: CLIP InfoNCE SimCLR
Closest papers by meaning
- Connect, Collapse, Corrupt: Learning Cross-Modal Tasks with Uni-Modal Data (ICLR 2024 · similarity 0.87)
- Extending Multi-modal Contrastive Representations (NeurIPS 2024 · similarity 0.87)
- On the Value of Cross-Modal Misalignment in Multimodal Representation Learning (NeurIPS 2025 · similarity 0.86)
- Cross the Gap: Exposing the Intra-modal Misalignment in CLIP via Modality Inversion (ICLR 2025 · similarity 0.86)
- The "Law'' of the Unconscious Contrastive Learner: Probabilistic Alignment of Unpaired Modalities (ICLR 2025 · similarity 0.86)
- Multi-modal contrastive learning adapts to intrinsic dimensions of shared latent variables (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.