Research map / Probabilistic and scientific modeling / Variational and generative sampling
Multimodal Variational Autoencoder: A Barycentric View
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Reframes multimodal VAE posterior aggregation as a barycenter problem, showing PoE and MoE are KL barycenters, and proposes Wasserstein-barycenter VAEs (WB-VAE and mixture MWB-VAE) evaluated on PolyMNIST, MNIST-SVHN-TEXT and bimodal CelebA.
Problem
Aggregating unimodal inference distributions into a joint posterior in multimodal VAEs is done with ad hoc PoE/MoE functions whose theoretical properties are poorly understood, and KL-based aggregation is asymmetric and unbounded, not defining a metric over distributions.
Approach
View the joint posterior as a barycenter of unimodal inference distributions under a chosen divergence; PoE and MoE are barycenters under reverse and forward KL, and using the squared 2-Wasserstein distance gives a geometry-preserving aggregation (closed form for Gaussians: weighted mean and weighted std for isotropic Gaussians), with a mixture-of-Wasserstein-barycenters variant.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add the mixture-of-Wasserstein-barycenters aggregation to WB-VAE (ablation) | WB-VAE · MNIST-SVHN-TEXT | linear classification accuracy (Avg.): 0.89 -> 0.97 | Helped |
| Add the mixture-of-Wasserstein-barycenters aggregation to WB-VAE (ablation) | WB-VAE · MNIST-SVHN-TEXT | conditional generation coherence (Avg.): 0.41 -> 0.75 | Helped |
| Add the mixture-of-Wasserstein-barycenters aggregation to WB-VAE (ablation) | WB-VAE · MNIST-SVHN-TEXT test set (joint X) | log-likelihood: -1785±7.4 -> -1890±1.7 | Hurt |
| Add the mixture-of-Wasserstein-barycenters aggregation to WB-VAE (ablation) | WB-VAE · CelebA | mean average precision (Avg.): 0.36 -> 0.40 | Helped |
Experimental setup
Datasets: PolyMNIST MNIST-SVHN-TEXT CelebA
Benchmarks: PolyMNIST MNIST-SVHN-TEXT CelebA
Metrics: linear classification accuracy conditional generation coherence log-likelihood mean average precision
Baselines: PoE-VAE MoE-VAE MoPoE-VAE
Builds on: MVAE MMVAE MoPoE-VAE optimal transport
Closest papers by meaning
- Efficient Modality Translation via Arbitrary Conditioning and Wasserstein Regularization (AAAI 2026 · similarity 0.90)
- Aggregation of Dependent Expert Distributions in Multimodal Variational Autoencoders (ICML 2025 · similarity 0.90)
- Unity by Diversity: Improved Representation Learning for Multimodal VAEs (NeurIPS 2024 · similarity 0.89)
- Mitigating Pervasive Modality Absence Through Multimodal Generalization and Refinement (AAAI 2025 · similarity 0.87)
- Amplifying Prominent Representations in Multimodal Learning via Variational Dirichlet Process (NeurIPS 2025 · similarity 0.87)
- When to Align, When to Predict: A Phase Diagram for Multimodal Learning (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.