Research map / Probabilistic and scientific modeling / Variational and generative sampling
A Unified View on Learning Unnormalized Distributions via Noise-Contrastive Estimation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Unifies MLE, MC-MLE, GlobalGISO, pseudo-likelihood/ISO-type estimators and conditional NCE under a generalized NCE framework (f-NCE, alpha-centered NCE, f-conditional NCE), showing f-CondNCE's variance diverges as noise vanishes, and giving finite-sample rates for bounded exponential families.
Problem
Learning unnormalized distributions (energy-based models / exponential families) with intractable normalizing constants; existing estimators (MLE, MC-MLE, GlobalGISO, pseudo-likelihood, interaction screening, conditional NCE) were proposed in separate communities without a unified comparison or fini
Approach
Introduces f-NCE via Bregman divergence minimization for density ratios, an alpha-centered NCE objective whose normalization makes it subsume MLE, MC-MLE and GlobalGISO as special cases, and f-conditional NCE with a channel that contrasts joint distributions; it also shows that small-noise conditional NCE does not reduce to score matching in finite samples.
Key results
- Shows alpha-CentNCE at alpha=0 equals GlobalGISO and at alpha=1 equals MLE (and MC-MLE with empirical noise), unifying these estimators.
- Establishes the first finite-sample O(n^{-1/2}) rate for regularized f-NCE, alpha-CentNCE and f-CondNCE estimators for bounded exponential families.
- Proves that f-CondNCE's empirical objective is dominated by an O(epsilon) noise term as epsilon to 0, so its variance blows up in the small-noise regime, contrary to prior score-matching claims.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use regularized f-NCE under bounded-statistic and bounded-parameter assumptions | f-NCE for exponential families · Bounded exponential families | parameter estimation error rate: O(n^{-1/2}) | Helped |
| Use regularized α-CentNCE under bounded-statistic and bounded-parameter assumptions | α-CentNCE for exponential families · Bounded exponential families | parameter estimation error rate: O(n^{-1/2}) | Helped |
| Use regularized f-CondNCE under bounded-statistic and bounded-parameter assumptions | f-CondNCE for exponential families · Bounded exponential families | parameter estimation error rate: O(n^{-1/2}) | Helped |
| Use f-log as the generator rather than another f in NCE | f-NCE · General unnormalized distributions | asymptotic covariance: \mathcal{V}_f^{\nce}\succeq \mathcal{V}_{f_{\log}}^{\nce} | Helped |
| Decrease the conditional-noise scale ε toward zero with a fixed number of conditional samples | f-CondNCE · Small-noise conditional estimation | asymptotic covariance: \approx \frac{1}{\epsilon^2} \mathbb{E}_{\pdata(x)}[\nabla_x\psi(x)\nabla_x\psi(x)^\intercal] | Hurt |
| Treat empirical f-CondNCE at small ε as equivalent to score matching | f-CondNCE · Finite-sample small-noise regime | estimator behavior | Hurt |
| Increase the number K of conditional samples while using small ε | Conditional NCE · Synthetic 1D Gaussian, N=10^4 | empirical derivative agreement | Helped |
Surprising findings
- Conditional NCE with small noise does not converge to score matching in the finite-sample regime; its variance diverges as the noise scale shrinks unless the number of conditional samples is large.
Limitations
- Finite-sample guarantees require bounded sufficient statistics and bounded parameter space, which excludes e.g. Gaussian models.
- Empirical evaluation is only a preliminary simulation (1D Gaussian and a small appendix experiment), not high-dimensional.
- Sample complexity constants grow exponentially in the parameter radius and dimension.
Experimental setup
Metrics: finite-sample convergence rate asymptotic variance
Baselines: MLE MC-MLE GlobalGISO score matching pseudo likelihood conditional NCE
Builds on: noise-contrastive estimation GlobalGISO conditional NCE score matching Negahban et al. regularized M-estimator theory
Closest papers by meaning
- Analyzing and Improving the Optimization Landscape of Noise-Contrastive Estimation (ICLR 2022 · similarity 0.91)
- Learning Unnormalized Statistical Models via Compositional Optimization (ICML 2023 · similarity 0.90)
- Conditional Noise-Contrastive Estimation of Unnormalised Models (ICML 2018 · similarity 0.89)
- Near-Optimality of Contrastive Divergence Algorithms (NeurIPS 2024 · similarity 0.87)
- Adaptive Estimation and Learning under Temporal Distribution Shift (ICML 2025 · similarity 0.85)
- Information Theoretic Learning for Diffusion Models with Warm Start (NeurIPS 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.