Research map / Probabilistic and scientific modeling / Variational and generative sampling
Unity by Diversity: Improved Representation Learning for Multimodal VAEs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MMVM VAE, a multimodal VAE whose data-dependent mixture-of-experts prior over unimodal posteriors softly aligns modality-specific latents, instead of hard-sharing an aggregated posterior.
Problem
Multimodal VAEs that aggregate unimodal encoders into a shared joint posterior impose hard constraints, causing trade-offs between reconstruction quality, latent representation quality, and coherence of cross-modal generation.
Approach
Replace the joint posterior aggregation with a data-dependent mixture-of-experts prior built from all unimodal posteriors of a sample; each modality keeps its own encoder and decoder, and the KL to this prior reduces to a Jensen-Shannon divergence between unimodal posteriors (soft alignment of positive pairs).
Key results
- On MIMIC-CXR, MMVM VAE reaches 73.1% average AUROC across labels, above the supervised baseline's 70.5% and the best VAE baseline (PoE 70.4%, MoPoE 70.3%).
- On hippocampal recordings, MMVM is the only method besides MoPoE with odor separation in latent space, and MoPoE has higher reconstruction loss.
- MMVM consistently outperforms independent, AVG, MoE, PoE, MoPoE across benchmark datasets in latent representation and coherence versus reconstruction trade-off.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the MMVM data-dependent mixture-of-experts prior instead of the strongest reported VAE baseline | MIMIC-CXR latent representations · MIMIC-CXR — All labels | AUROC (%): 70.4 ±8.3 -> 73.1 ±8.8 | Helped |
| Use the MMVM data-dependent mixture-of-experts prior instead of the strongest reported VAE baseline | MIMIC-CXR latent representations · MIMIC-CXR — No Finding | AUROC (%): 77.2 ±0.2 -> 78.7 ±0.4 | Helped |
| Use the MMVM data-dependent mixture-of-experts prior instead of the strongest reported VAE baseline | MIMIC-CXR latent representations · MIMIC-CXR — Cardiomegaly | AUROC (%): 76.7 ±1.9 -> 79.6 ±0.9 | Helped |
| Use the MMVM data-dependent mixture-of-experts prior instead of the strongest reported VAE baseline | MIMIC-CXR latent representations · MIMIC-CXR — Edema | AUROC (%): 82.1 ±1.2 -> 85.3 ±1.0 | Helped |
| Use the MMVM data-dependent mixture-of-experts prior instead of the strongest reported VAE baseline | MIMIC-CXR latent representations · MIMIC-CXR — Lung Lesion | AUROC (%): 61.3 ±2.1 -> 63.6 ±0.7 | Helped |
| Use the MMVM data-dependent mixture-of-experts prior instead of the strongest reported VAE baseline | MIMIC-CXR latent representations · MIMIC-CXR — Pneumonia | AUROC (%): 57.8 ±0.7 -> 59.5 ±0.7 | Helped |
| Use MMVM rather than aggregation-based or independent VAEs | Multimodal benchmark VAEs · Translated PolyMNIST, bimodal CelebA, and CUB | Latent representation classification and coherence | Helped |
| Use MMVM rather than previous VAE methods | Hippocampal neural activity representations · Five-rat memory experiment | Latent representation classification and conditional generation coherence | Helped |
Surprising findings
- Coherent conditional generation arises even though the MMVM decoder never sees samples from another modality during training.
- The MMVM lateral-view representation outperforms the PoE frontal-view representation on cardiomegaly, suggesting soft sharing enhances the weaker modality.
Limitations
- The data-dependent prior precludes unconditional random generation of samples.
- Scalability to larger multimodal datasets is untested because training requires large compute.
- Experiments use relatively small benchmark and specialized datasets.
Experimental setup
Datasets: translated PolyMNIST bimodal CelebA CUB image-captions hippocampal neural activity (5 rats) MIMIC-CXR MIMIC-CXR-JPG
Benchmarks: translated PolyMNIST bimodal CelebA CUB image-captions hippocampal neural activity MIMIC-CXR
Metrics: linear classifier accuracy on latent representations conditional generation coherence reconstruction loss AUROC FID
Baselines: independent VAEs AVG multimodal VAE MoE multimodal VAE PoE multimodal VAE MoPoE MMVAE+ supervised ResNet classifier
Builds on: VAMP-prior beta-VAE MMVAE MoPoE MMVAE+
Closest papers by meaning
- On the Limitations of Multimodal VAEs (ICLR 2022 · similarity 0.91)
- MMVAE+: Enhancing the Generative Quality of Multimodal VAEs without Compromises (ICLR 2023 · similarity 0.91)
- Mitigating Modality Collapse in Multimodal VAEs via Impartial Optimization (ICML 2022 · similarity 0.90)
- Multimodal Gaussian Mixture Variational Autoencoder with Consistency Regularizations (AAAI 2026 · similarity 0.90)
- Aggregation of Dependent Expert Distributions in Multimodal Variational Autoencoders (ICML 2025 · similarity 0.90)
- Learning Multimodal VAEs through Mutual Supervision (ICLR 2022 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.