Research map / Probabilistic and scientific modeling / Variational and generative sampling
Sobolev Space Regularised Pre Density Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SOSREP, a non-parametric pre-density estimator that minimizes a log-likelihood term plus a Sobolev RKHS norm regularizer, with a sampling-approximated SDO kernel, natural gradients, and Fisher-divergence hyperparameter tuning, achieving second-best ADBench anomaly detection.
Problem
Non-parametric density estimation in higher dimensions that is non-negative, integrable, expressive, and computationally feasible, with interpretable inductive bias; log-likelihood cannot tune hyperparameters for unnormalized pre-densities.
Approach
Regularize a Sobolev RKHS norm of f while fitting -mean log f^2(x_i), yielding unnormalized pre-densities f^2; approximate the SDO kernel by Fourier-feature sampling, optimize with natural gradients (which better preserve positivity) from non-negative initialization, and tune the bandwidth a via Fisher divergence score matching.
Key results
- SOSREP ranks second best among ~16 anomaly detection algorithms on ADBench (47 datasets) by both mean AUC-ROC and rank.
- On duplicate-anomaly X5 setting, SOSREP achieves average AUC-ROC of 71.6, a lead of about 4% over the closest contender.
- Standard gradient descent yields about 70% negative values in worst 5 of 50 initializations on the 'shuttle' dataset, while natural gradient keeps negative fraction unchanged.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use the SOSREP estimator with the SDO kernel | ADBench anomaly detection · ADBench, 47 tabular datasets | rank: 2nd best overall | Helped |
| use SOSREP with the SDO kernel for duplicate anomalies | ADBench anomaly detection · ADBench X5 duplicate anomalies | AUC-ROC: 71.6; a lead of 4% over the closest contender | Helped |
| use the SDO kernel instead of Gaussian or Laplace kernels (ablation) | SOSREP · ADBench anomaly detection | AUC-ROC | Helped |
| use natural gradient instead of standard gradient descent (ablation) | SOSREP optimization · ADBench, 15 randomly chosen datasets | fraction of negative values: about 70% negative values in the 5 worst cases for standard gradient; unchanged by natural gradient on shuttle | Helped |
| initialize alpha with non-negative values | SOSREP gradient descent · SOSREP optimization | performance | Helped |
Surprising findings
- For two-cluster block model with equal sizes, SOSREP density ratio is independent of inter-cluster correlation beta, unlike KDE.
- Standard gradient descent from non-negative initialization can amplify negative values in the solution, which natural gradient avoids.
Limitations
- Does not integrate to 1 for finite N; normalization constant estimation left out of scope.
- Non-convex optimization; standard gradient descent performs poorly without careful initialization and natural gradient.
- Performance with Gaussian and Laplace kernels is generally competitive but SDO kernel is better; reasons unclear.
Experimental setup
Datasets: ADBench
Benchmarks: ADBench
Metrics: AUC-ROC average rank fraction of negative values
Baselines: KDE (Gaussian kernel) KDE (Laplace kernel) Isolation Forest ADBench AD algorithms (15 methods)
Builds on: kernel density estimation random Fourier features score matching regularized maximum likelihood (1d spline SOSREP-type estimators)
Closest papers by meaning
- Efficient Nonparametric Smoothness Estimation (NeurIPS 2016 · similarity 0.87)
- Black-box density function estimation using recursive partitioning (ICML 2021 · similarity 0.85)
- Density Ratio Estimation with Doubly Strong Robustness (ICML 2024 · similarity 0.84)
- ConjNorm: Tractable Density Estimation for Out-of-Distribution Detection (ICLR 2024 · similarity 0.84)
- Bounds on $L_p$ Errors in Density Ratio Estimation via $f$-Divergence Loss Functions (ICLR 2025 · similarity 0.83)
- Sobolev Regularized MMD Gradient Flow (NeurIPS 2026 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.