Research map / Vision-language and multimodal / Cross-modal learning
The "Law'' of the Unconscious Contrastive Learner: Probabilistic Alignment of Unpaired Modalities
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Theoretically shows that comparing contrastive representations of unpaired modalities (A, C) via a shared bridge modality B recovers the probability ratio p(C|A)/p(C), and proposes a LogSumExp Monte Carlo estimator for when the direct-comparison "law" fails.
Problem
Contrastive encoders trained on paired modalities A-B and B-C are often used to compare unpaired modalities A and C by direct representation similarity, but it is unclear when this is probabilistically justified and when it fails.
Approach
Treat contrastive critics as density ratios and integrate out the bridge modality; under uniform-on-hypersphere (normalized dot product) or Gaussian (L2 critic) marginal assumptions, this integral has a closed form that is a monotone function of the direct similarity, yielding the "Law" of the Unconscious Contrastive Learner. Otherwise, a Monte Carlo LogSumExp over bridge-modality samples estimate
Key results
- On AudioSet zero-shot audio-visual retrieval with CLIP and CLAP, LogSumExp reaches 62% Recall@10 versus 14% for direct comparison.
- With LanguageBind, direct evaluation gets 70% Recall@10 while LogSumExp gets 58%, with the gap attributed to too few Monte Carlo samples.
- In language-conditioned navigation, LogSumExp improves success rates by roughly 20%-30% over direct comparison across environments.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use LogSumExp marginalization instead of direct image-audio embedding comparison | CLIP/CLAP · AudioSet | Recall@10: 14% -> 62% | Helped |
| Use direct evaluation instead of LogSumExp marginalization | LanguageBind · AudioSet | recall: 58% -> 70% | Helped |
| Use LogSumExp instead of direct evaluation | ImageBind with CLAP · Vision-Language alignment | Recall@1: 32.4% ± 1.5% -> 31.4% ± 1.7% | No clear effect |
| Use LogSumExp instead of direct evaluation | ImageBind with CLIP · Audio-Language alignment | Recall@1: 29.1% ± 1.7% -> 29.4% ± 3.3% | No clear effect |
| Use LogSumExp marginalization instead of direct representation comparison | Language-conditioned navigation · PointMaze environments | success rate: 20%-$30% | Helped |
| Increase the number of sampled intermediate embeddings (ablation) | LogSumExp with ImageBind and LanguageBind · AudioSet | Recall@1 accuracy: 1 to 500,000 | Helped |
| Use audio as the intermediate modality (ablation) | LogSumExp modality bridging · Vision-Language alignment | Recall@1: 31.4% ± 1.7% vs 32.4% ± 1.5% | No clear effect |
| Use images as the intermediate modality (ablation) | LogSumExp modality bridging · Audio-Language alignment | Recall@1: 29.4% ± 3.3% vs 29.1% ± 1.7% | No clear effect |
| Violate conditional independence by adding shared disturbance to A and C unseen by B (ablation) | Synthetic modality-bridging methods · Synthetic retrieval | retrieval accuracy | Hurt |
Surprising findings
- Direct comparison of normalized dot-product representations performs well even when the uniform-marginal assumption appears violated, suggesting the assumption may not be necessary.
- Direct comparison of unpaired CLIP image and CLAP audio encoders performs near chance, showing the heuristic can fail without a shared bridge.
Limitations
- Relies on conditional independence A ⊥ C | B and on the uniform or Gaussian marginal assumptions for the closed-form law; violations degrade performance.
- Monte Carlo approximation needs enough bridge-modality samples; LanguageBind gap closes only with many samples or larger intermediate sets.
- Does not give a practical test for when users should prefer direct comparison over the Monte Carlo method.
Experimental setup
Datasets: AudioSet AudioSet ontology synthetic Gaussian data PointMaze-based language navigation environments
Benchmarks: AudioSet synthetic retrieval benchmark PointMaze language navigation
Metrics: Recall@1 Recall@10 success rate Kolmogorov-Smirnov p-value
Baselines: Direct comparison of representations Ground truth oracle trained on (A,C) pairs LanguageBind ImageBind CLIP CLAP
Builds on: InfoNCE contrastive representations as density ratios contrastive reinforcement learning ImageBind LanguageBind
Closest papers by meaning
- Extending Multi-modal Contrastive Representations (NeurIPS 2024 · similarity 0.89)
- Aligning Multimodal Representations through an Information Bottleneck (ICML 2025 · similarity 0.88)
- Connect, Collapse, Corrupt: Learning Cross-Modal Tasks with Uni-Modal Data (ICLR 2024 · similarity 0.88)
- On the Value of Cross-Modal Misalignment in Multimodal Representation Learning (NeurIPS 2025 · similarity 0.88)
- Guiding Cross-Modal Representations with MLLM Priors via Preference Alignment (NeurIPS 2025 · similarity 0.88)
- Aligning Visual Contrastive learning models via Preference Optimization (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.