Research map / Image generation / Fast diffusion sampling

Feature Information Dynamics in Diffusion

NeurIPS 2026 · Image generation › Fast diffusion sampling · approach: Diffusion models as perception tools

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces feature information dynamics, an I-MMSE-based framework that localizes along the noise schedule when a feature (frequency band, class, mask, Canny edge) is generated, and uses chained decomposition to confirm spectral autoregression in pixel diffusion and compare pixel, SDVAE, VAVAE, and RAE representations.

Problem

Diffusion models are widely observed to generate coarse structure before fine detail, but this has only been described qualitatively; there is no general quantitative, representation-agnostic account of when features are generated along the diffusion trajectory.

Approach

Using the I-MMSE identity, the rate of change of mutual information between a feature and the noisy observation equals half the gap between unconditional and feature-conditional MMSE, giving a feature information density over log-SNR; a chained decomposition over nested feature bundles separates incremental from shared information.

Key results

• Best FID over 200 XL epochs: RAE 8.39 < VAVAE 21.16 < SDVAE 30.83 < pixel 78.51 under matched recipe.
• On CIFAR-10, chained frequency bands show a monotone coarse-to-fine progression in log-SNR (spectral autoregression).
• In pixel diffusion mask information peaks at lower log-SNR than class identity; only RAE shows a strict class→mask→Canny ordering of density peaks.

Surprising findings

• In pixel-space diffusion, mask structure enters earlier and more strongly than class identity, challenging a semantic-first account.
• Fitting unconditional and conditional MMSE curves separately and differencing gave meaningless mutual information estimates (around -10^5 nats).

Limitations

• Chained attribution is order-dependent; only one canonical class→mask→Canny order plus one alternative was tested.
• Approximation by trained denoisers means measured gaps depend on optimization quality; only four representations and one conditioning chain studied.
• Association between ordering and convergence speed is correlational, not causal; larger hierarchies require many cumulative conditional models.

Experimental setup

Datasets: ImageNet-256 (SR95 subset) CIFAR-10 MNIST

Benchmarks: ImageNet-256 FID CIFAR-10 MMSE/information density MNIST MMSE/information density

Metrics: FID MMSE feature information density peak log-SNR mutual information

Baselines: pixel diffusion (JiT-L/16) SDVAE (SiT-XL/2) VAVAE (LightningDiT-XL/1) RAE (DiTDH-XL)

Builds on: I-MMSE identity flow matching spectral autoregression Representation Autoencoders (RAE) VA-VAE

Closest papers by meaning

• Interpretable Diffusion via Information Decomposition (ICLR 2024 · similarity 0.90)
• Tracing the Roots: Leveraging Temporal Dynamics in Diffusion Trajectories for Origin Attribution (NeurIPS 2025 · similarity 0.90)
• Adapting to Unknown Low-Dimensional Structures in Score-Based Diffusion Models (NeurIPS 2024 · similarity 0.89)
• Understanding Representation Dynamics of Diffusion Models via Low-Dimensional Modeling (NeurIPS 2025 · similarity 0.88)
• Conservation Laws for Diffusion Models (NeurIPS 2026 · similarity 0.88)
• Critical windows: non-asymptotic theory for feature emergence in diffusion models (ICML 2024 · similarity 0.88)