Research map / Image generation / Fast diffusion sampling
On the Feature Learning in Diffusion Models
Andi Han, Wei Huang, Yuan Cao, Difan Zou
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Theoretically and empirically compares feature learning of diffusion models (DDPM loss) versus classification in a two-layer quadratic CNN on a two-patch signal-plus-noise data model, showing diffusion learns balanced signal and noise while classification learns dominant features depending on SNR.
Problem
How feature learning emerges during diffusion model training and how it differs from supervised classification training, in particular whether diffusion models learn signal and noise features in a different balance.
Approach
Uses a multi-patch data model with two orthogonal signal vectors and a two-layer quadratic-activation CNN, analyzing gradient descent on the DDPM loss (weight-shared autoencoder-style network) versus logistic loss classification; shows diffusion reaches a stationary point with signal-to-noise learning ratio Theta(n·SNR^2), whereas classification exhibits a sharp phase transition in n·SNR^2.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace classification training with the DDPM denoising objective | Two-layer network on the synthetic dataset · Synthetic dataset, low SNR (n · SNR² = 0.75) | balance of signal and noise feature learning | Improved |
| Replace classification training with the DDPM denoising objective | Two-layer network on the synthetic dataset · Synthetic dataset, high SNR (n · SNR² = 6.75) | balance of signal and noise feature learning | Improved |
| Replace classification training with the DDPM denoising objective | Two-layer network on Noisy-MNIST · Noisy-MNIST, low SNR (SNR̃ = 0.1) | balance of signal and noise feature learning | Improved |
| Increase Noisy-MNIST signal scale from SNR̃ = 0.1 to SNR̃ = 0.5 (ablation) | Classification model · Noisy-MNIST | signal versus noise feature learning | Mixed |
| Increase diffusion time step from t = 0.2 to t = 0.8 (ablation) | Diffusion model on Noisy-MNIST · Noisy-MNIST, SNR̃ = 0.1 and SNR̃ = 0.5 | balance of signal and noise feature learning | No clear effect |
| Extend the task from two classes to 10 classes (ablation) | Diffusion model and classification model on Noisy-MNIST · 10-class Noisy-MNIST (SNR̃ = 0.1) | signal versus noise feature learning | Mixed |
Surprising findings
- Classification at low SNR reaches 100% training accuracy yet only about 50% test accuracy because it primarily fits noise.
Limitations
- Analysis restricted to a two-patch data model, two-layer quadratic networks, and a single diffusion time step with decoupled per-time-step weights.
- Diffusion model converges only to a stationary point rather than a global minimum of the loss.
- Experiments are small-scale (synthetic and MNIST subsets) without large-scale diffusion models.
Experimental setup
Datasets: MNIST synthetic multi-patch data (Gaussian noise patches)
Benchmarks: synthetic multi-patch data Noisy-MNIST
Metrics: signal learning magnitude (max inner product with signal vec noise learning magnitude training loss test accuracy
Baselines: supervised classification with logistic loss
Builds on: DDPM feature learning theory of two-layer CNNs (Allen-Zhu & Li style) benign overfitting analysis
Closest papers by meaning
- Understanding diffusion models requires rethinking (again) generalization (NeurIPS 2026 · similarity 0.90)
- Understanding Representation Dynamics of Diffusion Models via Low-Dimensional Modeling (NeurIPS 2025 · similarity 0.90)
- Feature Information Dynamics in Diffusion (NeurIPS 2026 · similarity 0.90)
- Learning Diffusion Models with Flexible Representation Guidance (NeurIPS 2025 · similarity 0.90)
- Generalization Dynamics of Linear Diffusion Models (NeurIPS 2026 · similarity 0.89)
- Neural Flow Diffusion Models: Learnable Forward Process for Improved Diffusion Modelling (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
