Research map / Learning theory and algorithms / Kernels and regression theory

Algorithm- and Data-Dependent Generalization Bounds for Diffusion Models

NeurIPS 2025 · Learning theory and algorithms › Kernels and regression theory · approach: Information-theoretic generalization bounds · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Derives algorithm- and data-dependent generalization bounds for score-based diffusion models by decomposing the score approximation error into score matching loss, a data-dependent diffusion gap, and a score generalization gap, then applying SGLD/ADAM-oriented bounds (gradient norms, topological complexity) validated on Gaussian mixture, MNIST, flowers and butterflies.

Problem

Existing statistical guarantees for score-based generative models use approximation theory and ignore the training set and the optimization algorithm, so they cannot explain the effect of optimizer hyperparameters on generalization of the generated distribution.

Approach

Decompose the score approximation error into the explicit score matching loss, a data-dependent diffusion gap (empirical vs true data forward process), and a score generalization gap; bound the gap with existing algorithm-dependent learning theory (SGLD gradient-norm bound and topological/weighted-lifetime-sum bounds), yielding a KL bound of order O(n^{-1/2}) plus training loss.

Key results

• The averaged gradient-norm quantity b<||g||^2> correlates well with the generalization gap for ADAM on butterflies and flowers.
• The high-probability bound gives a KL rate of O(n^{-1/2}) for fixed step size h, with worst-case rate n^{-O(1/d)} after optimizing h.
• The gradient-norm bound (SGLD) overestimates the observed generalization gap in magnitude but correlates with it, especially for high inverse temperature.

Surprising findings

• The data-dependent diffusion gap has nonpositive expectation over the dataset, so it can be dropped in expectation but matters for high-probability bounds.
• The gradient-norm-based SGLD bound overestimates the observed generalization gap by a large margin, attributed to unknown sub-Gaussian constant or implicit regularization.

Limitations

• The subgaussian assumption on the loss is not verified in practice and the SGLD theory does not directly cover SGD/ADAM; a heuristic substitution of beta by b/eta is used.
• Experiments are low-dimensional or on small image datasets with limited training runs; well-trained models across many settings were not feasible.
• Bounds are coarse, do not explicitly model the noise schedule, and the bounded-support assumption is strong.

Experimental setup

Datasets: Gaussian mixture (4D, 9 components) MNIST flowers butterflies

Benchmarks: Gaussian mixture test Wasserstein-2 MNIST FID butterflies FID flowers FID

Metrics: Wasserstein-2 FID score generalization gap (train/test epsilon-loss) correlation with complexity measures

Baselines: SGLD ADAM

Builds on: SGLD generalization bounds (Mou et al. style) topological generalization bounds KL convergence bounds for score-based models (Chen et al.) exponential integrator sampler

Closest papers by meaning

• Dimension-free Score Matching and Time Bootstrapping for Diffusion Models (NeurIPS 2025 · similarity 0.90)
• Generalization Dynamics of Linear Diffusion Models (NeurIPS 2026 · similarity 0.89)
• Beyond Log-Concavity and Score Regularity: Improved Convergence Bounds for Score-Based Generative Models in W2-distance (ICML 2025 · similarity 0.89)
• Evaluating the design space of diffusion-based generative models (NeurIPS 2024 · similarity 0.89)
• Approximation and Generalization Abilities of Score-based Neural Network Generative Models for Sub-Gaussian Distributions (NeurIPS 2025 · similarity 0.89)
• Global Well-posedness and Convergence Analysis of Score-based Generative Models via Sharp Lipschitz Estimates (ICLR 2025 · similarity 0.89)