Research map / Probabilistic and scientific modeling / Variational and generative sampling
SD-KDE: Score-Debiased Kernel Density Estimation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SD-KDE, which shifts each sample one step along an estimated score and then applies KDE with a modified bandwidth to cancel leading-order bias, improving MISE convergence from n^{-4/(d+4)} to n^{-8/(d+8)}.
Problem
Standard kernel density estimation suffers from a bias-variance trade-off controlled by bandwidth, with bias dominating for highly variable densities; the classical MISE rate is n^{-4/(d+4)}.
Approach
Each data point is moved by a single score-function step with step size h^2/2 before standard Gaussian KDE with bandwidth h, which cancels the leading O(h^2) bias term (a one-step Langevin/Euler-Maruyama-style shift), and the method remains effective with noisy score estimates.
Key results
- On 1D Gaussian mixtures with n=5x10^4, SD-KDE with the true score has roughly an order of magnitude smaller MISE than Silverman KDE on average over 50 seeds.
- Fitted MISE decay slopes for SD-KDE were -0.85 (mixture 1) and -0.93 (mixture 3), close to the theoretical -8/9 ≈ -0.89 rate.
- For n=100, SD-KDE beat Silverman KDE on all 100 seeds for mixtures 1 and 2, and on 95% of seeds for mixture 3.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add score-based sample adjustment and modified bandwidth | KDE · 1D Gaussian mixtures | MISE: order of magnitude smaller | Helped |
| Add score-based sample adjustment and modified bandwidth using oracle scores | Silverman KDE · 2D spiral | MISE | Helped |
| Use a diffusion-model estimated score instead of the oracle score (ablation) | SD-KDE · 2D spiral | MISE | Mixed |
| Add noise to the score estimate, up to σ=4 (ablation) | SD-KDE · 1D Gaussian mixtures | MISE: σ=4 | Helped |
| Estimate the score using Silverman KDE instead of assuming the true score is known (ablation) | SD-KDE · 1D Gaussian mixtures | MISE | Helped |
| Apply a second score-debiased iteration (ablation) | Iterated SD-KDE · 1D Gaussian mixture | KL divergence: more than a factor of three | Helped |
| Continue iterating beyond the second score-debiased correction (ablation) | Iterated SD-KDE · 1D Gaussian mixture | KL divergence | Hurt |
| Use the true score instead of a learned score (ablation) | SD-KDE · 2D mixture of Gaussians | MISE: nearly an order of magnitude | Helped |
Surprising findings
- With a DDPM-estimated score on 2D spiral data, SD-KDE performed only comparably to Silverman KDE, which the authors attribute to diffusion model training difficulties rather than the method.
Limitations
- Theoretical guarantees require an exact score oracle, which is typically unavailable in practice.
- Performance with a learned score (diffusion model) on 2D data was only comparable to Silverman KDE.
- Only one-step correction is analyzed; multi-step/higher-order discretizations are left to future work.
Experimental setup
Datasets: MNIST 1D Gaussian mixtures 1D Laplace mixtures 2D spiral distribution 2D mixture of Gaussians
Benchmarks: MNIST 1D Gaussian mixtures 1D Laplace mixtures 2D spiral distribution 2D mixture of Gaussians
Metrics: mean integrated squared error KL divergence
Baselines: Silverman KDE Emp-SD-KDE
Builds on: kernel density estimation Silverman bandwidth rule score matching denoising diffusion probabilistic models
Closest papers by meaning
- ConjNorm: Tractable Density Estimation for Out-of-Distribution Detection (ICLR 2024 · similarity 0.85)
- Kernel Debiased Plug-in Estimation: Simultaneous, Automated Debiasing without Influence Functions for Many Target Parameters (ICML 2024 · similarity 0.85)
- Overcoming Saturation in Density Ratio Estimation by Iterated Regularization (ICML 2024 · similarity 0.84)
- Debiased Distribution Compression (ICML 2024 · similarity 0.84)
- Kernel Regression in Structured Non-IID Settings: Theory and Implications for Denoising Score Learning (NeurIPS 2025 · similarity 0.84)
- Dynamic Similarity Graph Construction with Kernel Density Estimation (ICML 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.