Research map / Vision-language and multimodal / Cross-modal learning
Understanding the Emergence of Multimodal Representation Alignment
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Empirically studies when representation alignment emerges between independently trained unimodal models and whether alignment predicts task performance, varying redundancy/uniqueness and heterogeneity on synthetic data, vision-language models, and MultiBench.
Problem
Alignment between independently trained unimodal representations is known to emerge with scale (Platonic Representation Hypothesis), but it is unclear when alignment emerges implicitly and whether it reliably predicts multimodal task performance.
Approach
Characterizes datasets along interactions (redundancy vs uniqueness) and heterogeneity, showing alignment's maximum achievable level falls and its correlation with performance weakens as uniqueness grows, so alignment is not universally beneficial and depends on data.
Key results
- On AVMNIST (high redundancy), alignment strongly correlates with performance (Vision-Audio correlations 0.887 and 0.723).
- On affective computing MultiBench datasets, alignment-performance correlations are weak and often negative (e.g., MOSEI Vision-Text -0.093 and Audio-Text -0.641).
- As uniqueness increases, maximum achievable alignment decreases and the alignment-performance correlation drops toward zero, with negative correlations for U > 3 in synthetic data.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase task-relevant uniqueness between modalities | independently trained unimodal encoders · Synthetic datasets | maximum alignment | Hurt |
| Increase heterogeneity between modalities | independently trained unimodal encoders · Synthetic datasets | maximum alignment | Hurt |
| Increase encoder capacity relative to transformation depth | independently trained encoders · Synthetic datasets with low uniqueness | alignment | Helped |
| Increase encoder capacity relative to transformation depth | independently trained encoders · Synthetic datasets with high uniqueness | alignment | No clear effect |
| Increase task-relevant uniqueness | alignment-performance relationship · Synthetic datasets | alignment-performance correlation | Mixed |
| Increase model capacity | unimodal models · Synthetic datasets across uniqueness levels | task performance | Helped |
| Increase task-relevant uniqueness | DINOv2 and language-model representations · Wikipedia caption dataset | alignment-performance relationship | Hurt |
| Use higher alignment while increasing the CLIP alignment-loss weight | CLIP vision and language models · MM-IMDb classes with lower alignment-performance slopes | classification performance | Hurt |
| Use higher alignment while increasing the CLIP alignment-loss weight | CLIP vision and language models · MM-IMDb classes with higher alignment-performance slopes | classification performance | Helped |
| Increase alignment | multimodal models · AVMNIST | performance correlation: +0.887 Vision; +0.723 Audio | Helped |
| Increase alignment | multimodal models · MOSEI, MOSI, URFUNNY, and MUStARD | performance correlation: MOSEI: -0.223, -0.172, -0.093, -0.482, -0.145, -0.641; MOSI: -0.014, 0.313, 0.116, -0.413, 0.403, -0.386; URFUNNY: -0.33 | Mixed |
Surprising findings
- Alignment is negatively correlated with performance for a significant proportion of tasks with high uniqueness.
- Model depth and performance remain positively correlated even when alignment-performance correlation vanishes.
Limitations
- Synthetic experiments use simplified binary features and MLP/transformer encoders rather than large-scale models.
- Uniqueness levels for real vision-language data rely on GPT-4o-generated captions and a heuristic percentage scale.
- MM-IMDb experiments use a limited subset (1024 examples per split) and only 10 genre classes.
Experimental setup
Datasets: synthetic binary data Wikipedia image-caption dataset GPT-4o-generated uniqueness captions MultiBench (MOSEI, MOSI, URFUNNY, MUStARD, AVMNIST) MM-IMDb
Benchmarks: MultiBench MM-IMDb synthetic redundancy-uniqueness datasets
Metrics: CKA mutual KNN Spearman correlation bits-per-byte loss classification accuracy
Baselines: CLIP loss finetuning
Builds on: Platonic Representation Hypothesis CLIP Centered Kernel Alignment mutual nearest neighbors alignment
Closest papers by meaning
- On the Value of Cross-Modal Misalignment in Multimodal Representation Learning (NeurIPS 2025 · similarity 0.91)
- Aligning Multimodal Representations through an Information Bottleneck (ICML 2025 · similarity 0.90)
- When to Align, When to Predict: A Phase Diagram for Multimodal Learning (NeurIPS 2026 · similarity 0.90)
- Implicit Multimodal Alignment: On the Generalization of Frozen LLMs to Multimodal Inputs (NeurIPS 2024 · similarity 0.89)
- The Alignment Illusion in Multimodal Large Language Models (NeurIPS 2026 · similarity 0.89)
- Asymmetric Reinforcing Against Multi-Modal Representation Bias (AAAI 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.