Research map / Vision-language and multimodal / Cross-modal learning
Amplifying Prominent Representations in Multimodal Learning via Variational Dirichlet Process
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DPMM, a Dirichlet-process mixture of per-modality Gaussian components trained with stochastic variational inference, to amplify prominent intra-modal features while aligning modalities, and uses the learned marginals to impute missing modalities in EHR/CXR/notes clinical prediction.
Problem
Multimodal fusion must align modality distributions without shrinking modality-specific features, and must handle missing modalities in clinical data where existing alignment and imputation (zeros, means) weaken representations.
Approach
Model each modality's feature distribution as a mixture of multivariate Gaussians whose weights come from a stick-breaking Dirichlet process over modality-component pairs, so the richer-gets-richer property amplifies prominent features; optimize via stochastic variational inference and sample from the learned marginals to impute missing modalities.
Key results
- On MIMIC-IV totally matched IHM, DPMM reaches AUROC 0.826 and AUPR 0.482 versus best baseline DrFuse 0.816/0.446.
- On MIMIC-III partially matched IHM, DPMM reaches AUROC 0.860 vs. 0.854 for DAFT.
- On CMU-MOSI, DPMM achieves MAE 1.09 vs. 1.11 for the best baseline MedFuse.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DPMM instead of the strongest baseline | MIMIC-III totally matched IHM · MIMIC-III IHM | AUROC: 0.835 -> 0.854 | Helped |
| Use DPMM instead of the strongest baseline | MIMIC-IV totally matched IHM · MIMIC-IV IHM | AUPR: 0.446 -> 0.482 | Helped |
| Use DPMM instead of the strongest baseline | CMU-MOSI · CMU-MOSI | MAE: 1.11 -> 1.09 | Helped |
| Use DPMM instead of the strongest baseline | POM · POM | Accuracy: 0.358 -> 0.359 | Helped |
| add DP alignment (ablation) | DPMM without alignment on MIMIC-IV · MIMIC-IV totally matched IHM | AUROC: 0.809 -> 0.826 | Helped |
| add gradient-preserving sampling (ablation) | DPMM without GPS on MIMIC-IV · MIMIC-IV totally matched IHM | AUROC: 0.855 -> 0.860 | Helped |
| add the fusion module (ablation) | DPMM without fusion on MIMIC-IV · MIMIC-IV totally matched IHM | AUROC: 0.857 -> 0.860 | Helped |
| use DP alignment loss instead of cosine loss (ablation) | MIMIC-IV DPMM alignment · MIMIC-IV totally matched IHM | AUROC: 0.816 -> 0.826 | Helped |
| use DP alignment loss instead of KL loss (ablation) | MIMIC-IV DPMM alignment · MIMIC-IV totally matched IHM | AUROC: 0.820 -> 0.826 | Helped |
| add cross-modal alignment to DP amplification (ablation) | MIMIC-IV DPMM · MIMIC-IV totally matched IHM | AUROC: 0.816 -> 0.826 | Helped |
| use DP mixture weights instead of learnable weights (ablation) | MIMIC-IV DPMM · MIMIC-IV partially matched IHM | AUROC: 0.855 -> 0.860 | Helped |
| increase the maximum mixture count K (ablation) | DPMM · MIMIC-IV READM | AUROC | Helped |
Surprising findings
- Learnable mixture weights in place of DP weights gave lower performance (e.g., 0.821 vs 0.826 AUROC on totally matched IHM).
- Amplification alone already improves over the w/o DP baseline, but adding alignment gives further gains.
Limitations
- Uses diagonal covariance only; full covariance and Wishart priors are left for future work.
- Hierarchical DP variants not explored.
- Training time is higher than most baselines.
Experimental setup
Datasets: MIMIC-III MIMIC-IV MIMIC-CXR MIMIC-III NOTE CMU-MOSI POM
Benchmarks: MIMIC-III MIMIC-IV CMU-MOSI POM
Metrics: AUROC AUPR F1 MAE accuracy Corr
Baselines: MMTM DAFT Unified MedFuse DrFuse LMF TFN CLIP ALIGN CoMM PSA LSMT
Builds on: DrFuse MedFuse Dirichlet process mixture model stick-breaking construction
Closest papers by meaning
- MSP: Modality Self-Play for Training Multimodal LLMs Without Paired Cross-Modal Data (NeurIPS 2026 · similarity 0.89)
- Beyond Modality Fusion: Deep Ensembles for Multimodal Classification (NeurIPS 2026 · similarity 0.88)
- Fusion or Confusion? Multimodal Complexity Is Not All You Need (NeurIPS 2026 · similarity 0.88)
- Aligning Multimodal Representations through an Information Bottleneck (ICML 2025 · similarity 0.87)
- On the Value of Cross-Modal Misalignment in Multimodal Representation Learning (NeurIPS 2025 · similarity 0.87)
- When to Align, When to Predict: A Phase Diagram for Multimodal Learning (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.