Research map / Optimization / Implicit bias and generalization

Neural Network-Based Score Estimation in Diffusion Models: Optimization and Generalization

ICLR 2024 · Optimization › Implicit bias and generalization · approach: In-context learning as gradient descent · 3 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Analyzes gradient-descent training of a two-layer ReLU network for denoising score matching, proving minimax-optimal N^{-2/(d+3)} score estimation error via NTK kernel-regression coupling and an early-stopping rule for unbounded inputs.

Problem

Whether gradient-descent-trained neural networks can provably learn the score function in diffusion models, covering both optimization and generalization, given unbounded inputs, noisy DSM labels, vector-valued outputs and a time variable.

Approach

Parameterize the score via a score decomposition E[X0|Xt] so DSM becomes regression with noisy labels; show GD on a two-layer ReLU net tracks NTK kernel regression, and use an early-stopping rule extended to unbounded domains to get minimax-optimal sample complexity.

Key results

• Early-stopped GD-trained two-layer ReLU network achieves score estimation error Õ(N^{-2/(d+3)}), claimed minimax-optimal.
• Experiments on Credit Default tabular data show performance comparable to heavily tuned heuristic training schemes.

Limitations

• Network width must grow polynomially in N (m ~ poly(d) N^{26+...}), which is far from practical.
• Analysis uses bounded data support (Assumption on ||X0|| <= D) and a fixed second layer during training.

Experimental setup

Datasets: Credit Default dataset

Benchmarks: Credit Default dataset

Metrics: fidelity of generated tabular data

Baselines: heuristic diffusion training schemes

Builds on: neural tangent kernel denoising score matching score-based generative modeling with SDEs kernel early stopping

Closest papers by meaning

• What’s the score? Automated Denoising Score Matching for Nonlinear Diffusions (ICML 2024 · similarity 0.90)
• Nearest Neighbour Score Estimators for Diffusion Generative Models (ICML 2024 · similarity 0.90)
• Training Diffusion-based Generative Models with Limited Data (ICML 2025 · similarity 0.89)
• Generalization through variance: how noise shapes inductive biases in diffusion models (ICLR 2025 · similarity 0.89)
• Dimension-free Score Matching and Time Bootstrapping for Diffusion Models (NeurIPS 2025 · similarity 0.89)
• Minimax Optimality of Score-based Diffusion Models: Beyond the Density Lower Bound Assumptions (ICML 2024 · similarity 0.88)