Research map / Probabilistic and scientific modeling / Variational and generative sampling
Density Ratio Estimation with Conditional Probability Paths
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Conditional Time Score Matching (CTSM and vectorized CTSM-v) to learn the time score for density ratio estimation via conditional probability paths, giving faster training and comparable or better accuracy than TSM, with error bounds.
Problem
Density ratio estimation in high dimensions via probability paths requires estimating the time score, but existing Time Score Matching (TSM) is computationally expensive (double autodiff) and can be inaccurate, with unclear theoretical guarantees.
Approach
Introduce a conditioning variable z so the intractable marginal time score equals the expectation of the tractable conditional time score; regress on the conditional time score (CTSM), and vectorize it over dimensions (CTSM-v), avoiding the second-order derivatives of TSM, with a time score normalization weighting and importance sampling.
Key results
- On MNIST latent-space EBM, CTSM-v reaches approx. 1.26 BPD versus 1.30 for TSM while taking 58 ms per step vs 347 ms.
- On Gaussian density-ratio tasks, CTSM and CTSM-v take around 5 ms per iteration versus around 15 ms for TSM and improve accuracy over TSM for the same iterations.
- CTSM-v trained directly in MNIST pixel space reaches approx. 1.03 BPD where TSM was unstable.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use CTSM-v instead of TSM | MNIST latent EBM · MNIST latent | Approx. BPD: 1.30 -> 1.26 | Helped |
| Use CTSM-v instead of TSM | MNIST latent EBM · MNIST latent | Time per step: 347 ms -> 58 ms | Helped |
| Use CTSM-v instead of TSM | MNIST ambient pixel-space EBM · MNIST ambient | Approx. BPD | Helped |
| Use CTSM-v instead of TSM | MNIST ambient pixel-space EBM · MNIST ambient | Time per step: 1103 ms -> 142 ms | Helped |
| Use CTSM or CTSM-v instead of TSM | Density-ratio estimation for distant Gaussians · Distant Gaussians | Time per iteration: around $15$ms -> around $5$ms | Helped |
| Use CTSM or CTSM-v instead of TSM | Density-ratio estimation for distant Gaussians · Distant Gaussians | Estimation accuracy | Helped |
| Use CTSM-v instead of CTSM or TSM | High-dimensional Gaussian mutual-information estimation · High-dimensional Gaussians | Convergence speed | Helped |
| Use CTSM-v instead of CTSM or TSM under a fixed compute budget | High-dimensional Gaussian mutual-information estimation · High-dimensional Gaussians | Mutual-information estimation | Helped |
| Use CTSM-v instead of TSM (ablation) | MNIST time-score estimation · MNIST | Trainability | Helped |
| Replace linear time embeddings with Fourier feature embeddings (ablation) | CTSM-v U-Net for MNIST EBM · MNIST | Training stability and final performance | Helped |
Surprising findings
- Non-vectorized CTSM is essentially not trainable on MNIST, so vectorization is needed.
- Annealed importance sampling log-normalizer estimates vary widely (roughly -1100 to 650), making BPD unreliable for unnormalized models.
Experimental setup
Datasets: MNIST
Benchmarks: MNIST
Metrics: mean-squared error of log density ratio bits per dimension time per iteration mutual information estimation error
Baselines: TSM NCE TRE
Builds on: Time Score Matching denoising score matching flow matching Noise Contrastive Estimation
Closest papers by meaning
- Binary Losses for Density Ratio Estimation (ICLR 2025 · similarity 0.86)
- Overcoming Saturation in Density Ratio Estimation by Iterated Regularization (ICML 2024 · similarity 0.86)
- Density Ratio Estimation-based Bayesian Optimization with Semi-Supervised Learning (ICML 2025 · similarity 0.86)
- Projection Pursuit Density Ratio Estimation (ICML 2025 · similarity 0.86)
- Nonparametric Estimation of a Factorizable Density using Diffusion Models (NeurIPS 2026 · similarity 0.86)
- Dequantified Diffusion-Schrödinger Bridge for Density Ratio Estimation (ICML 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.