Research map / Learning theory and algorithms / Sample complexity
Approximation and Generalization Abilities of Score-based Neural Network Generative Models for Sub-Gaussian Distributions
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that ReLU-network score-based generative models trained on sub-Gaussian data achieve near-minimax score-matching and total-variation rates without Lipschitz-score or density lower-bound assumptions.
Problem
Theoretical approximation and generalization guarantees for neural-network score-based generative models (SGMs) estimating an unknown sub-Gaussian distribution from n samples, previously requiring strong regularity (Lipschitz score, density lower bounds).
Approach
Uses a regularized empirical (KDE-based) score surrogate, empirical Bayes smoothing to bound score matching loss by Hellinger distance with a polynomial-in-1/sigma bound, and a compact ReLU network construction whose size does not grow with n; bounds the excess risk via Bernstein's inequality and an epsilon-net since the surrogate is uniformly bounded.
Key results
- Score-matching error of ReLU DNN score estimator bounded by tilde-O(n^-1 t0^(-d/2)) for t0 > O(alpha^2 n^(-2/d) log n).
- TV distance of learned distribution at early stopping time bounded by tilde-O(alpha^(d/2) t0^(-d/4) n^(-1/2)).
- With early stopping t0 = n^(-2/(d+2s)), TV error for Sobolev/Besov densities is polylog(n) * n^(-s/(d+2s)), nearly minimax.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use a ReLU neural network to estimate scores from samples under a sub-Gaussian assumption | Score-based generative models · Sub-Gaussian distributions, t ∈ [t0, T] | score matching loss: t_0^{-d/2} n^{-1}\log^{\frac{d}{2}+4}n + t_0^{-1} n^{-1} \log n \cdot \log\frac{2}{\delta} | Helped |
| Use ReLU SGM with early stopping at t0 | Score-based generative models · Sub-Gaussian distributions at early stopping time t0 | total variation distance: \alpha^{d/2} t_0^{-d/4} n^{-1/2} \log^{\frac{d}{4}+2}n + t_0^{-1/2} n^{-1/2} \log^{1/2}n \cdot \sqrt{\log(2/\delta)} | Helped |
| Apply early stopping with t0 = n^{-\frac{2}{d+2s}} for a Sobolev-class density | ReLU SGM · Sobolev-class density | total variation distance: \mathrm{polylog}(n) n^{-\frac{s}{d+2s}} | Helped |
| Apply early stopping with t0 = n^{-\frac{2}{d+2s}} for a Besov-class density | ReLU SGM · Besov-class density on [0, 1]^d | total variation distance: n^{-\frac{s}{d+2s}} | Helped |
Surprising findings
- The paper reports a missing h^(-d/2) factor in a prior proof (Oko et al.), implying that the prior sigma^(-d-2) scaling should be sigma^(-2)-type rather than sigma^(-d-2).
Limitations
- Bounds still suffer from the curse of dimensionality; no manifold or low-rank adaptation.
- Analysis relies on sub-Gaussian tails; extension to heavier tails left open.
- Rates restricted to smoothness 0 < s <= 2 due to linear isotropic Gaussian kernel estimator.
Experimental setup
Metrics: score matching loss total variation distance
Baselines: Lipschitz-score KDE estimator (Wibisono et al.) truncated KDE score estimator (Oko et al.) Holder/Besov DNN SGM analyses
Builds on: score-based generative modeling kernel score estimation (Zhang et al.) empirical Bayes score estimation
Closest papers by meaning
- Algorithm- and Data-Dependent Generalization Bounds for Diffusion Models (NeurIPS 2025 · similarity 0.89)
- Supervised Score-Based Modeling by Gradient Boosting (AAAI 2025 · similarity 0.87)
- Neural Network-Based Score Estimation in Diffusion Models: Optimization and Generalization (ICLR 2024 · similarity 0.86)
- Global Well-posedness and Convergence Analysis of Score-based Generative Models via Sharp Lipschitz Estimates (ICLR 2025 · similarity 0.86)
- Beyond Log-Concavity and Score Regularity: Improved Convergence Bounds for Score-Based Generative Models in W2-distance (ICML 2025 · similarity 0.86)
- Score-based generative models are provably robust: an uncertainty quantification perspective (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.