Research map / Probabilistic and scientific modeling / Variational and generative sampling
Aggregation of Dependent Expert Distributions in Multimodal Variational Autoencoders
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CoDE-VAE, a multimodal VAE that aggregates single-modality (expert) Gaussians via a Bayesian consensus of dependent experts with learned per-subset ELBO weights, avoiding independence assumptions of PoE/MoE.
Problem
Joint posterior approximation in multimodal VAEs: PoE and MoE assume independence between single-modality experts, which is overoptimistic, and MoE sub-sampling of modalities hurts likelihood and generative quality as modalities grow.
Approach
Model expert estimation errors as correlated Gaussians (correlation rho between experts) so the consensus posterior is a Bayesian Gaussian posterior that generalizes PoE; additionally learn per-subset ELBO weights pi_k via an entropy-based indicator variable, removing equal weighting and sub-sampling.
Key results
- On MNIST-SVHN-Text, CoDE-VAE gets the best unconditional coherence (0.51) and conditional coherence (0.82) among compared models, with conditional coherence 0.82 vs 0.79 for MoPoE-VAE.
- CoDE-VAE is the only model reaching unconditional FID similar to unimodal VAEs at 4 and 5 modalities on PolyMNIST.
- On CUB caption-to-image, CoDE-VAE achieves conditional coherence 0.75 with conditional FID 175.97, close to MMVAE+ (0.72, 164.94).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use CoDE-VAE instead of benchmark models | Multimodal VAE · MNIST-SVHN-Text | Generative coherence and joint log-likelihood | Helped |
| Use CoDE-VAE instead of benchmark models | Multimodal VAE · MNIST-SVHN-Text | Conditional coherence: 0.74 -> 0.82 | Helped |
| Use CoDE-VAE instead of benchmark models | Multimodal VAE · CUB | Conditional coherence: 0.72 -> 0.75 | Helped |
| Learn ELBO weights π and model expert dependence with ρ instead of equal weights and independent experts (ablation) | CoDE-VAE · MNIST-SVHN-Text | Overall performance across β values | Helped |
| Learn ELBO weights π instead of using equal ELBO weights (ablation) | CoDE-VAE · PolyMNIST | FID | Helped |
| Model expert dependence by cross-validating ρ instead of setting ρ=0 (ablation) | CoDE-VAE · PolyMNIST | FID | Helped |
| Increase expert correlation ρ (ablation) | CoDE-VAE · PolyMNIST | Average joint log-likelihood | Helped |
| Train without modality sub-sampling | CoDE-VAE · PolyMNIST, 4- and 5-modality training | Unconditional FID | Helped |
Surprising findings
- Learned weights pi_k are higher for subsets with more modalities, and the ELBO dominating optimization on MNIST-SVHN-Text is the 3-modality subset.
- The consensus mean can have negative weights on experts when the correlation is high (rho above about 0.5), leaning toward the more certain expert.
Limitations
- Cross-validating rho is computationally costly for large data.
- Computational cost grows as O(2^M-1) in number of subsets, though feasible for 5 modalities.
- Unclear how the framework generalizes to other prior or likelihood choices within the Bayesian formulation.
Experimental setup
Datasets: MNIST-SVHN-Text PolyMNIST CUB Image-Caption
Benchmarks: MNIST-SVHN-Text PolyMNIST CUB Image-Caption
Metrics: conditional coherence unconditional coherence FID log-likelihood estimation linear classification accuracy
Baselines: MVAE MMVAE mmJSD MoPoE-VAE MVTCAE MMVAE+
Builds on: multimodal VAE product of experts mixture of experts MoPoE-VAE beta-VAE
Closest papers by meaning
- On the Limitations of Multimodal VAEs (ICLR 2022 · similarity 0.92)
- MMVAE+: Enhancing the Generative Quality of Multimodal VAEs without Compromises (ICLR 2023 · similarity 0.91)
- Variational Mixture-of-Experts Autoencoders for Multi-Modal Deep Generative Models (NeurIPS 2019 · similarity 0.90)
- Unity by Diversity: Improved Representation Learning for Multimodal VAEs (NeurIPS 2024 · similarity 0.90)
- Multimodal Variational Autoencoder: A Barycentric View (AAAI 2025 · similarity 0.90)
- Multimodal Generative Models for Scalable Weakly-Supervised Learning (NeurIPS 2018 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.