Research map / Learning theory and algorithms / Kernels and regression theory
Statistical and Geometrical properties of the Kernel Kullback-Leibler divergence
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Studies the kernel Kullback-Leibler (KKL) divergence defined via kernel covariance operators, proposes a regularized variant KKL_alpha well-defined for discrete distributions with disjoint supports, gives deviation and finite-sample bounds, a closed-form via Gram matrices, and a Wasserstein gradient descent scheme.
Problem
The original kernel KKL divergence is infinite when supports differ, so it cannot compare discrete distributions; its statistical properties and closed-form computation for empirical measures and optimization behavior were unexplored.
Approach
Regularize KKL by mixing the target with the source, KKL_alpha(p||q) = KKL(p||(1-alpha)q + alpha p), which makes it finite for any distributions; derive skewness bounds, finite-sample concentration, a Gram-matrix closed form and its first variation for Wasserstein gradient descent on particles.
Key results
- The regularized KKL deviates from the original by a bound linear in alpha (times |Tr(Sigma_p log Sigma_q)|), and it is monotonically decreasing in alpha.
- Empirical KKL_alpha concentrates around its population value faster for larger alpha, consistent with the finite-sample bounds.
- KKL gradient descent with L-BFGS recovers the three-rings target and the spiral shape, with MMD particles drifting off the target support while KKL and KALE follow it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| increase regularization parameter α | regularized KKL · Gaussian distributions; empirical KKL concentration | concentration speed | Helped |
| increase regularization parameter α | regularized KKL · fixed p and q distributions | KKL divergence | Hurt |
| use regularized KKL gradient descent instead of MMD gradient descent | gradient descent for transporting particles to a three-ring target · three-ring target | target support alignment | Helped |
| use KKL or KALE gradient descent instead of MMD gradient descent | gradient descent for transporting particles to a spiral target · spiral target | shape recovery speed | Helped |
Surprising findings
- MMD flow particles can leave the target rings because MMD is insensitive to support mismatch, whereas KKL follows the support closely.
Limitations
- Theoretical convergence guarantees for the KKL gradient flow are not established.
- Computational cost is O((n+m)^3) and reducing it is left to future work.
- KKL struggles to recover disconnected outliers in the spiral experiment.
Experimental setup
Datasets: synthetic Gaussian and exponential distributions three rings target heart-to-spiral shape transfer
Benchmarks: synthetic 2D shape transfer (three rings, heart to spiral) concentration experiments in dimension 10
Metrics: KKL deviation vs sample size convergence of particle flow to target
Baselines: MMD KALE
Builds on: Kernel Kullback-Leibler divergence (Bach 2022) KALE Maximum mean discrepancy Wasserstein gradient flow
Closest papers by meaning
- Decoupled Kullback-Leibler Divergence Loss (NeurIPS 2024 · similarity 0.88)
- Kernel Quantile Embeddings and Associated Probability Metrics (ICML 2025 · similarity 0.88)
- Connecting Jensen–Shannon and Kullback–Leibler Divergences: A New Bound for Representation Learning (NeurIPS 2025 · similarity 0.87)
- Stochastic variance-reduced Gaussian variational inference on the Bures-Wasserstein manifold (ICLR 2025 · similarity 0.86)
- Better Estimation of the Kullback--Leibler Divergence Between Language Models (NeurIPS 2025 · similarity 0.86)
- The Minimax Rate of HSIC Estimation for Translation-Invariant Kernels (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.