AI Paper Atlas

Research map / Image generation / Fast diffusion sampling

On the Feature Learning in Diffusion Models

Andi Han, Wei Huang, Yuan Cao, Difan Zou

ICLR 2025 · Image generation › Fast diffusion sampling · approach: Diffusion theory and design space · 12 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Theoretically and empirically compares feature learning of diffusion models (DDPM loss) versus classification in a two-layer quadratic CNN on a two-patch signal-plus-noise data model, showing diffusion learns balanced signal and noise while classification learns dominant features depending on SNR.

Problem

How feature learning emerges during diffusion model training and how it differs from supervised classification training, in particular whether diffusion models learn signal and noise features in a different balance.

Approach

Uses a multi-patch data model with two orthogonal signal vectors and a two-layer quadratic-activation CNN, analyzing gradient descent on the DDPM loss (weight-shared autoencoder-style network) versus logistic loss classification; shows diffusion reaches a stationary point with signal-to-noise learning ratio Theta(n·SNR^2), whereas classification exhibits a sharp phase transition in n·SNR^2.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
Replace classification training with the DDPM denoising objectiveTwo-layer network on the synthetic dataset · Synthetic dataset, low SNR (n · SNR² = 0.75)balance of signal and noise feature learningImproved
Replace classification training with the DDPM denoising objectiveTwo-layer network on the synthetic dataset · Synthetic dataset, high SNR (n · SNR² = 6.75)balance of signal and noise feature learningImproved
Replace classification training with the DDPM denoising objectiveTwo-layer network on Noisy-MNIST · Noisy-MNIST, low SNR (SNR̃ = 0.1)balance of signal and noise feature learningImproved
Increase Noisy-MNIST signal scale from SNR̃ = 0.1 to SNR̃ = 0.5 (ablation)Classification model · Noisy-MNISTsignal versus noise feature learningMixed
Increase diffusion time step from t = 0.2 to t = 0.8 (ablation)Diffusion model on Noisy-MNIST · Noisy-MNIST, SNR̃ = 0.1 and SNR̃ = 0.5balance of signal and noise feature learningNo clear effect
Extend the task from two classes to 10 classes (ablation)Diffusion model and classification model on Noisy-MNIST · 10-class Noisy-MNIST (SNR̃ = 0.1)signal versus noise feature learningMixed

Surprising findings

Limitations

Experimental setup

Datasets: MNIST synthetic multi-patch data (Gaussian noise patches)

Benchmarks: synthetic multi-patch data Noisy-MNIST

Metrics: signal learning magnitude (max inner product with signal vec noise learning magnitude training loss test accuracy

Baselines: supervised classification with logistic loss

Builds on: DDPM feature learning theory of two-layer CNNs (Allen-Zhu & Li style) benign overfitting analysis

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy