Research map / Vision-language and multimodal / Cross-modal learning
To Align or Not to Align: Strategic Multimodal Representation Alignment for Optimal Performance
Wanlong Fang, Tianle Zhang, Alvin Chan
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| increase explicit alignment weight lambda (ablation) | unimodal MLP encoders · Synthetic data, high redundancy (R=6, 8) | accuracy | Improved |
| increase explicit alignment weight lambda (ablation) | unimodal MLP encoders · Synthetic data, low redundancy (R=0, 2) | accuracy | Hurt |
| increase explicit alignment weight lambda (ablation) | unimodal MLP encoders · Synthetic data, moderate redundancy (R=4) | accuracy: peak around lambda 0.4-0.6, then declines | Mixed |
| increase alignment weight lambda for Vision encoder (ablation) | Vision encoder (Vision-Text pair) · CMU-MOSEI | accuracy | Improved |
| increase alignment weight lambda for Text encoder (ablation) | Text encoder (Vision-Text pair) · CMU-MOSEI | accuracy: peaks around lambda 0.75, slight drop after | Mixed |
| increase alignment weight lambda for Audio encoder (ablation) | Audio encoder (Vision-Audio pair) · AV-MNIST | accuracy | Improved |
| increase alignment weight lambda for Vision encoder (ablation) | Vision encoder (Vision-Audio pair) · AV-MNIST | accuracy | Hurt |
| increase alignment weight lambda for Vision encoder (ablation) | Vision encoder (Vision-Audio pair) · MUStARD | accuracy: 0.54 -> 0.62 | Improved |
| increase alignment weight lambda for Audio encoder (ablation) | Audio encoder (Vision-Audio pair) · MUStARD | accuracy: 0.58 -> 0.66 | Improved |
Closest papers by meaning
- Understanding the Emergence of Multimodal Representation Alignment (ICML 2025 · similarity 0.94)
- With Limited Data for Multimodal Alignment, Let the STRUCTURE Guide You (NeurIPS 2025 · similarity 0.92)
- On the Value of Cross-Modal Misalignment in Multimodal Representation Learning (NeurIPS 2025 · similarity 0.91)
- Aligning Multimodal Representations through an Information Bottleneck (ICML 2025 · similarity 0.91)
- UNIALIGN: Scaling Multimodal Alignment within One Unified Model (CVPR 2025 · similarity 0.91)
- DeepAlign: Mitigating Modality Conflict through Modality-Specific Alignment (CVPR 2026 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
