Research map / Probabilistic and scientific modeling / Variational and generative sampling
Multiple Hypothesis Dropout: Estimating the Parameters of Multi-Modal Output Distributions
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Multiple Hypothesis (MH) Dropout with a stochastic winner-take-all loss to estimate means and variances of multi-modal output distributions (MoM), and extends VQVAE/VQGAN latent posteriors with variance estimates (MH-VQ) for better codebook efficiency.
Problem
Multiple-output prediction needs to represent several plausible outputs with uncertainty; mixture density networks are unstable in high dimensions, and multiple choice learning gives only point estimates. VQ autoencoders represent only cluster modes, not their spread, requiring large codebooks.
Approach
Convert a single network into a multiple-output function by treating its dropout subnetworks as shared-parameter hypotheses trained with a stochastic WTA loss over random subsets; the spread of subnetwork predictions estimates variance, and this is used both in a mixture of such networks (MoM) and as a variance branch in VQ latent spaces (MH-VQ).
Key results
- MH-VQVAE with 32 codes outperforms VQVAE2 with 256 codes on precision and recall on FashionMNIST.
- MH-VQ reduces required codebook entries up to 4x for VQVAE-2, 8x for VQGAN on FashionMNIST, and 16x on CelebA64 (MH-VQGAN with 16 vs VQGAN with 256).
- On the inverse sine wave, the MoM fits the tri-modal structure while FFN and MC-dropout mixtures fail.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MH-VQVAE instead of VQVAE-2 | VQVAE-2 · FashionMNIST | precision | Helped |
| Use MH-VQVAE instead of VQVAE-2 | VQVAE-2 · FashionMNIST | recall | Helped |
| Use MH-VQGAN instead of VQGAN | VQGAN · FashionMNIST, CelebA64, and ImageNet64 | precision | Helped |
| Use MH-VQGAN instead of VQGAN | VQGAN · FashionMNIST, CelebA64, and ImageNet64 | recall | Helped |
| Use MH-VQ instead of the corresponding VQ model | VQVAE-2 and VQGAN · FashionMNIST, CelebA64, and ImageNet64 validation samples | FID | Helped |
| Use a subset ratio between 0.5 and 0.7 during Stochastic WTA training (ablation) | MH Dropout network · Toy multi-point dataset | SDD | Helped |
| Use a subset ratio under 0.4 during Stochastic WTA training (ablation) | MH Dropout network · Toy multi-point dataset | SDD | Hurt |
| Use a subset ratio above 0.8 during Stochastic WTA training (ablation) | MH Dropout network · Toy multi-point dataset | SDD | Hurt |
| Use MC dropout instead of Stochastic WTA with an appropriate subset ratio (ablation) | MH Dropout network · Toy multi-point dataset | SDD | Hurt |
| Use parameter sharing in place of independent predictor parameters (ablation) | Multiple-output network · Toy multi-point dataset | targets learned: 3 of 5 -> each of the 5 | Helped |
| Reduce the subset size during stochastic hypothesis sampling (ablation) | MH Dropout network · Toy multi-point dataset | hypothesis cluster spread | Mixed |
| Use MoM with MH Dropout instead of a mixture of FFNs with MC dropout | Mixture of FFNs · Inverse sine wave | mean and variance fit | Helped |
Surprising findings
- MC dropout variance estimates from a single network do not generalize to multi-output settings and collapse to the mean of targets.
- Lower subset ratios in stochastic WTA approximate MC dropout and collapse hypotheses to an erroneous single-point average.
Limitations
- Experiments on the VQ side reuse the VQVAE-2 encoder and decoder across models due to GPU memory constraints.
- Connection between predictive variance and uncertainty estimation is left for future work.
- Stochastic WTA results are sensitive to the subset ratio, with a narrow optimal range.
Experimental setup
Datasets: FashionMNIST CelebA ImageNet 64x64
Benchmarks: FashionMNIST CelebA64 ImageNet64
Metrics: FID F1/8 (precision) F8 (recall) MSE LPIPS standard deviation distance
Baselines: MC dropout vanilla WTA / multiple choice learning mixture mixture density network VQVAE-2 VQGAN
Builds on: multiple choice learning mixture density networks Monte Carlo dropout VQ-VAE-2 VQGAN
Closest papers by meaning
- Dropout as a Structured Shrinkage Prior (ICML 2019 · similarity 0.83)
- Stochastic Modified Equations and Dynamics of Dropout Algorithm (ICLR 2024 · similarity 0.83)
- Dropout-Based Rashomon Set Exploration for Efficient Predictive Multiplicity Estimation (ICLR 2024 · similarity 0.82)
- Data Mixture Optimization: A Multi-fidelity Multi-scale Bayesian Framework (NeurIPS 2025 · similarity 0.82)
- Generalized Tensor Decomposition for Understanding Multi-Output Regression under Combinatorial Shifts (NeurIPS 2024 · similarity 0.82)
- Solving Sparse \& High-Dimensional-Output Regression via Compression (NeurIPS 2024 · similarity 0.81)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.