Research map / Probabilistic and scientific modeling / Variational and generative sampling

Generic Unsupervised Optimization for a Latent Variable Model With Exponential Family Observables

NeurIPS 2024 · Probabilistic and scientific modeling › Variational and generative sampling · approach: Latent variable marginalization and estimators

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes EF-MCA, a latent variable model with binary latents, maximum (rather than sum) superposition, and exponential family observables, and derives generic EM M-step updates with the same functional form for all regular exponential family distributions, combined with EVO variational E-steps.

Problem

Parameter optimization for latent variable models whose observables follow general (regular) exponential family distributions is challenging, since existing methods are tailored to specific distributions or restricted to one-parameter families.

Approach

Replace linear summation of binary latents with a weighted maximum to set the observable mean, parameterize each exponential family distribution with L dictionary matrices, and show via EM that the M-step updates take one generic form (weighted averages of sufficient statistics T_l(y) using posterior expectations of the max-selection indicator) for any regular exponential family distribution.

Key results

• Derives M-step updates W^(l)_dh = sum_n E[A_dh] T_l(y_d^(n)) / sum_n E[A_dh] that share one functional form across all regular exponential family distributions.
• Shows that the ELBO second derivative at fixed points is non-positive, supporting that the fixed points are maxima.
• Numerically verifies the analytical updates on several observable distributions and demonstrates applications in variance learning, noise type estimation and denoising.

Limitations

• Binary latents with a fixed maximum non-linearity may be restrictive in some settings.
• Closed-form inverse mapping of the natural parameters is not available for all distributions (e.g., Beta, Gamma), so the M-step then requires numerical approximation.
• Fixed-point updates are not strictly proven to converge to maxima; second-order analysis only gives partial evidence.

Experimental setup

Datasets: natural image patches

Metrics: log-likelihood parameter recovery

Baselines: EF-SC G-MCA Poisson PCA

Builds on: Maximal Causes Analysis Evolutionary Variational Optimization Exponential family sparse coding Expectation maximization

Closest papers by meaning

• Tractable Expected Information Gains for Exponential Family Posteriors (ICML 2026 · similarity 0.82)
• A Computationally Efficient Method for Learning Exponential Family Distributions (NeurIPS 2021 · similarity 0.82)
• Efficient preconditioned stochastic gradient descent for estimation in latent variable models (ICML 2023 · similarity 0.82)
• SUMO: Unbiased Estimation of Log Marginal Probability for Latent Variable Models (ICLR 2020 · similarity 0.82)
• Learning Additive Exponential Family Graphical Models via $\ell_{2,1}$-norm Regularized M-Estimation (NeurIPS 2016 · similarity 0.82)
• A Unified View on Learning Unnormalized Distributions via Noise-Contrastive Estimation (ICML 2025 · similarity 0.80)