Research map / Vision-language and multimodal / Cross-modal learning
On the Value of Cross-Modal Misalignment in Multimodal Representation Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Formalizes cross-modal misalignment in image-text data via selection and perturbation biases in a latent variable model and proves almost-sure local block-identifiability of unbiased semantic variables at global minima of an alignment-entropy surrogate, validated with simulations, MPI3D-Complex, Causal3DIdent, and an OpenCLIP zero-shot case study.
Problem
Multimodal contrastive learning assumes image-text pairs share the same semantics, but real captions omit or distort semantic content (cross-modal misalignment); it is unclear how this affects which semantics learned representations capture and whether misalignment helps or hurts.
Approach
Model misalignment with a latent variable model containing selection bias (which semantic variables appear in text) and perturbation bias (which appear altered), and show that population global minimizers of a symmetric alignment-minus-marginal-entropy surrogate depend only on the unbiased semantic block and locally identify it almost surely.
Key results
- Under the independent-latent simulation, unbiased semantic variables are predicted with R^2 near 1 while selection-omitted variables have R^2 near 0.
- On MPI3D-Complex, misaligned semantic variables have low nonlinear-probe MCC under both selection and perturbation bias, while unbiased semantics are predicted with generally high MCC.
- OpenCLIP ViT-B/32 and ViT-L/14 zero-shot macro-F1 is higher on frequently captioned groups (Animal, Object) than under-captioned groups (Trait, Emot.); LAION-400M average coverage ranges from 0.0003% (Stere.) to 2.16% (Color).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| increase semantic coverage in captions during pretraining | MMCL representations · Numerical simulation, in-distribution regression | regression performance | Helped |
| apply appropriately chosen selection or perturbation bias | MMCL representations · Numerical simulation, OOD classification | classification performance | Helped |
| omit semantic variables from text through selection bias | MMCL representations · Numerical simulation, semantic identification | R²: R^2 \approx 0 | Hurt |
| preserve a semantic variable in text without perturbing it | MMCL representations · Numerical simulation, semantic identification | R²: R^2 \approx 1 | Helped |
| introduce selection or perturbation bias | MPI3D-Complex learned representations · MPI3D-Complex | MCC | Hurt |
| retain a semantic variable without selection or perturbation bias | MPI3D-Complex learned representations · MPI3D-Complex | MCC | Helped |
| select or preserve a semantic variable without perturbing it | Causal3DIdent learned image representations · Causal3DIdent | R² or MCC | Helped |
| omit or perturb a semantic variable in text | Causal3DIdent learned image representations · Causal3DIdent | R² or MCC | Hurt |
| include correlated semantic factors in text | Causal3DIdent learned image representations · Causal3DIdent | R² or MCC | Helped |
| use a model pretrained on captions with higher concept coverage | OpenCLIP ViT-B/32 and ViT-L/14 · Zero-shot evaluation on concept groups | macro-F1 | Mixed |
| use OpenCLIP ViT-B/32 or ViT-L/14 pretrained on LAION-400M | OpenCLIP · Zero-shot evaluation on concept groups | macro-F1 | Mixed |
Surprising findings
- Misaligned semantic variables can remain partially predictable from the learned representation through statistical dependence on the retained semantic block.
- Appropriately chosen selection or perturbation biases can improve OOD classification performance in the simulations.
Limitations
- Theory holds only for population global minimizers of a surrogate, not the practical InfoNCE loss, and does not guarantee unique global recovery.
- Discrete-latent and real-data experiments go beyond the continuous-variable theoretical assumptions.
- OpenCLIP case study is descriptive and does not isolate caption coverage from concept prevalence or visual difficulty.
Experimental setup
Datasets: MPI3D-Complex Causal3DIdent LAION-400M synthetic Gaussian latent data
Benchmarks: MPI3D-Complex Causal3DIdent synthetic simulation
Metrics: R^2 MCC macro-F1 accuracy
Baselines: CLIP
Builds on: CLIP InfoNCE block-identifiability Darmois construction Causal3DIdent
Closest papers by meaning
- Understanding the Emergence of Multimodal Representation Alignment (ICML 2025 · similarity 0.91)
- What to align in multimodal contrastive learning? (ICLR 2025 · similarity 0.90)
- MIDAS: Misalignment-based Data Augmentation Strategy for Imbalanced Multimodal Learning (NeurIPS 2025 · similarity 0.90)
- Aligning Multimodal Representations through an Information Bottleneck (ICML 2025 · similarity 0.90)
- Extending Multi-modal Contrastive Representations (NeurIPS 2024 · similarity 0.90)
- Asymmetric Reinforcing Against Multi-Modal Representation Bias (AAAI 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.