Research map / Probabilistic and scientific modeling / Variational and generative sampling
Momentum Particle Maximum Likelihood
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Momentum Particle Descent (MPD), a momentum-enriched Wasserstein-type gradient flow on the extended space (parameters x probability measures) for type II MLE of latent variable models, proves exponential convergence in continuous time and discretizes it into a particle algorithm.
Problem
Maximum likelihood estimation of latent variable models via minimizing the free energy over parameters and distributions; existing particle gradient descent (PGD) is a first-order method that is suboptimal compared with accelerated methods.
Approach
Adds Nesterov/underdamped-Langevin-style momentum to both the parameter and latent-particle components of the free energy gradient flow, yielding a damped Hamiltonian McKean-Vlasov system whose particle discretization (MPD) uses an exponential integrator with a gradient correction.
Key results
- On MNIST, MPD reaches FID 47.9 ± 0.6 versus 104.1 ± 1.2 for PGD and 45.9 ± 0.8 for ABP (ABP lowest).
- On CIFAR-10, MPD achieves FID 93.2 ± 1.5, the best or second-best with VI (94.2 ± 6.7) and ABP (97.0 ± 8.0), versus PGD at 106.2 ± 4.8.
- Continuous-time momentum-enriched flow is proven to converge exponentially in the free energy under a log-Sobolev inequality and K-Lipschitz gradient assumptions.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace PGD with MPD | VAE with VampPrior · MNIST | FID: 104.1 \pm 1.2 -> 47.9 \pm 0.6 | Helped |
| Replace PGD with MPD | VAE with VampPrior · CIFAR-10 | FID: 106.2 \pm 4.8 -> 93.2 \pm 1.5 | Helped |
| Replace MPD with ABP | VAE with VampPrior · MNIST | FID: 47.9 \pm 0.6 -> 45.9 \pm 0.8 | Hurt |
| Replace MPD with ABP | VAE with VampPrior · CIFAR-10 | FID: 93.2 \pm 1.5 -> 97.0 \pm 8.0 | Hurt |
| Use the exponential integrator instead of the NAG-like integrator for the parameter component (ablation) | MPD · Toy hierarchical model | optimization performance | Helped |
| Add gradient correction to the parameter and particle updates (ablation) | MPD · Toy hierarchical model | stability | Helped |
| Add momentum to both parameter and particle components rather than enriching only one component (ablation) | PGD · ToyHM(10, 12) | parameter convergence | Helped |
| Add momentum to both parameter and particle components rather than enriching only one component (ablation) | PGD · 1-d VAE density estimation on a two-component Gaussian mixture | empirical Wasserstein-1 | Helped |
| Use MPD rather than PGD | latent variable model fitting · Experiments | performance | Helped |
Surprising findings
- ABP (multi-step overdamped Langevin) matches or slightly beats MPD on MNIST FID.
- Variational inference performs well on CIFAR but falls short on MNIST.
Limitations
- Momentum parameters (eta, gamma for theta and x) require tuning, with no systematic tuning method; the momentum coefficient heuristic is used instead.
- Convergence proof relies on a log-Sobolev inequality and K-Lipschitz gradient assumptions that hold only in limited settings (rigorous approaches limited to simple targets).
Experimental setup
Datasets: MNIST CIFAR-10
Benchmarks: MNIST CIFAR-10 ToyHM
Metrics: FID parameter estimation error empirical Wasserstein-1
Baselines: PGD ABP Short Run amortized VI
Builds on: particle gradient descent underdamped Langevin dynamics Nesterov accelerated gradient Wasserstein gradient flow Alternating backpropagation
Closest papers by meaning
- Learning Latent Variable Models via Jarzynski-adjusted Langevin Algorithm (NeurIPS 2025 · similarity 0.87)
- Particle Semi-Implicit Variational Inference (NeurIPS 2024 · similarity 0.86)
- Parametric model reduction of mean-field and stochastic systems via higher-order action matching (NeurIPS 2024 · similarity 0.86)
- To discretize continually: Mean shift interacting particle systems for Bayesian inference (NeurIPS 2026 · similarity 0.85)
- GAD-PVI: A General Accelerated Dynamic-Weight Particle-Based Variational Inference Framework (AAAI 2024 · similarity 0.85)
- Sampling in Unit Time with Kernel Fisher-Rao Flow (ICML 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.