Research map / Probabilistic and scientific modeling / Variational and generative sampling
Sampling Multimodal Distributions with the Vanilla Score: Benefits of Data-Based Initialization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proves that early-stopped Langevin dynamics initialized at the empirical distribution of samples, run on a vanilla score estimate, provably samples mixtures of strongly log-concave distributions.
Problem
Vanilla score matching is known to fail for multimodal distributions because it misweights components; the question is whether the vanilla score can nonetheless be used to sample multimodal distributions.
Approach
Run Langevin dynamics with an L2-accurate vanilla score estimate, initialized at the empirical distribution of samples (data-based initialization) and stopped early before mixing, so that the conditional law recovers the mixture weights; the analysis uses a new log-Sobolev bound for well-connected mixtures.
Key results
- Theorem gives total variation guarantee d_TV(hat mu, mu) <= epsilon_TV for LMC with early stopping at time T initialized from M = Omega(p_*^{-2} eps^{-4} K^4 log(K/eps) log(K/tau)) samples, provided score error is small enough.
- Log-Sobolev constant of a connected well-overlapping mixture is bounded by C_{|I},p_*} / delta times max component LSI constant (new result).
- Simulations: in 1D, early-stopped Langevin (T=200) recovers the two-Gaussian ground truth, while run to stationarity (T=infinity) does not.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Initialize Langevin dynamics from empirical data samples and stop early | Vanilla-score Langevin sampling · Mixture of two Gaussians, 1D simulation | distribution fit | Helped |
| Run vanilla-score Langevin dynamics until mixing instead of stopping early | Vanilla score matching · Mixture of two Gaussians, 1D simulation | distribution fit | Hurt |
| Run Langevin dynamics for a moderate rather than large number of iterations | Empirically initialized Langevin dynamics with an approximate learned score · 32-dimensional mixture of Gaussians | within-component mixing | Mixed |
Surprising findings
- Initializing Langevin at the centers of mixture components instead of samples fails in high dimensions, since the dynamics cannot cross between shells (exp(Omega(d)) time).
- Running the dynamics to stationarity (vanilla score matching) fails to recover the correct mixture weights, while early stopping succeeds.
Limitations
- Dependence on the number of mixture components is likely suboptimal; the result requires mixtures of log-concave components with bounded number of components for polynomial dependence.
- Requires a good L2 score estimate, which is hard to obtain far from the data support; the analysis assumes a known component structure bounded by a ball of radius R.
Experimental setup
Datasets: synthetic mixtures of Gaussians
Metrics: total variation distance (theoretical bound) visual comparison of KDE densities
Baselines: vanilla score matching run to stationarity empirical distribution initialization at mixture centers
Builds on: score matching contrastive divergence annealed score matching
Closest papers by meaning
- Learned Reference-based Diffusion Sampler for multi-modal distributions (ICLR 2025 · similarity 0.87)
- Dimension-free Score Matching and Time Bootstrapping for Diffusion Models (NeurIPS 2025 · similarity 0.86)
- One-Step Diffusion Distillation through Score Implicit Matching (NeurIPS 2024 · similarity 0.86)
- Sampling by averaging: A multiscale approach to score estimation (NeurIPS 2025 · similarity 0.86)
- Nearest Neighbour Score Estimators for Diffusion Generative Models (ICML 2024 · similarity 0.85)
- Training Diffusion-based Generative Models with Limited Data (ICML 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.