Research map / Vision-language and multimodal / Cross-modal learning
Cross-Modal Alignment via Variational Copula Modelling
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces CMCM, a copula-driven multimodal learning framework that models each modality's features as a Gaussian mixture and aligns them via a variational copula joint distribution, which also enables sampling-based imputation of missing modalities on MIMIC data.
Problem
Aligning and fusing heterogeneous modality representations (EHR time series, chest X-ray images, clinical notes) into a joint distribution, and generating unbiased representations when some modalities are missing.
Approach
Interpret a copula as an alignment tool: model each modality's latent features with a K-component Gaussian mixture as the marginal, couple them with a copula (Gumbel/Frank/Gaussian) trained via stochastic variational inference with copula log-likelihood added to the ELBO, and sample from learned marginals to impute missing modalities.
Key results
- On MIMIC-IV totally matched IHM, CMCM reaches 0.827 AUROC and 0.492 AUPR, versus 0.818 AUROC for the best baseline DrFuse.
- On MIMIC-III partially matched IHM, CMCM reaches 0.856 AUROC and 0.510 AUPR, outperforming the best baseline by 1.5% AUPR as the paper states.
- In the tri-modal MIMIC-IV setting, CMCM reaches 0.824 AUROC on IHM versus 0.808 for DrFuse.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use copula alignment instead of cosine alignment (ablation) | CMCM · MIMIC-IV matched, IHM | AUROC: 0.820 -> 0.827 | Helped |
| use copula alignment instead of KL alignment (ablation) | CMCM · MIMIC-IV matched, IHM | AUROC: 0.826 -> 0.827 | Helped |
| add copula alignment (ablation) | CMCM · MIMIC-IV matched, IHM | AUROC: 0.809 -> 0.827 | Helped |
| add copula alignment (ablation) | CMCM · MIMIC-IV partially matched, IHM | AUROC: 0.855 -> 0.858 | Helped |
| add gradient-preserving sampling (ablation) | CMCM · MIMIC-IV partially matched, IHM | AUROC: 0.858 -> 0.858 | No clear effect |
| add fusion module (ablation) | CMCM · MIMIC-IV matched, IHM | AUROC: 0.811 -> 0.827 | Helped |
| add fusion module (ablation) | CMCM · MIMIC-IV partially matched, IHM | AUROC: 0.860 -> 0.858 | Hurt |
| use Gumbel rather than Frank copula (ablation) | CMCM · MIMIC-IV matched, IHM | AUROC: 0.827 -> 0.825 | No clear effect |
| use Gaussian rather than Frank copula (ablation) | CMCM · MIMIC-IV matched, IHM | AUROC: 0.827 -> 0.827 | No clear effect |
| use CMCM instead of DrFuse | MIMIC-III matched IHM prediction · MIMIC-III matched, IHM | AUROC: 0.835 -> 0.854 | Helped |
| use CMCM instead of DrFuse | MIMIC-IV matched IHM prediction · MIMIC-IV matched, IHM | AUROC: 0.818 -> 0.827 | Helped |
| use CMCM instead of DAFT | MIMIC-III partially matched IHM prediction · MIMIC-III partially matched, IHM | AUPR: 0.495 -> 0.510 | Helped |
Surprising findings
- Copula alignment performance is broadly robust to the choice of copula family (Gumbel, Frank, Gaussian), with the best family varying by task.
Limitations
- Learning the copula parameter with a neural network may be insufficient because the joint log-likelihood is not convex.
- Evaluated only on healthcare EHR/imaging/notes datasets from MIMIC; generalization to other multimodal domains is untested.
Experimental setup
Datasets: MIMIC-III MIMIC-IV MIMIC-CXR MIMIC-III NOTE
Benchmarks: MIMIC-III IHM MIMIC-III READM MIMIC-IV IHM MIMIC-IV READM
Metrics: AUROC AUPR
Baselines: MMTM DAFT Unified MedFuse DrFuse LSMT Interleaved
Builds on: MedFuse DrFuse Sklar's theorem
Closest papers by meaning
- When to Align, When to Predict: A Phase Diagram for Multimodal Learning (NeurIPS 2026 · similarity 0.87)
- Amplifying Prominent Representations in Multimodal Learning via Variational Dirichlet Process (NeurIPS 2025 · similarity 0.87)
- On the Value of Cross-Modal Misalignment in Multimodal Representation Learning (NeurIPS 2025 · similarity 0.86)
- AlignMamba: Enhancing Multimodal Mamba with Local and Global Cross-modal Alignment (CVPR 2025 · similarity 0.85)
- Modalities Contribute Unequally: Enhancing Medical Multi-modal Learning through Adaptive Modality Token Re-balancing (ICML 2025 · similarity 0.84)
- Jointly Modeling Inter- & Intra-Modality Dependencies for Multi-modal Learning (NeurIPS 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.