Research map / Image generation / Fast diffusion sampling

Manifolds, Random Matrices and Spectral Gaps: The geometric phases of generative diffusion

ICLR 2025 · Image generation › Fast diffusion sampling · approach: Diffusion theory and design space · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Analyzes the Jacobian spectrum of diffusion-model score functions to detect latent manifold dimensionality via spectral gaps, using a linear-manifold random-matrix/replica analysis and a random energy model view of memorization, and shows estimated dimensionality grows with dataset size.

Problem

How the latent manifold dimensionality of generative diffusion models evolves over diffusion time and with training set size, and how memorization reduces it.

Approach

Uses spectral gaps in the singular values of the smoothed score Jacobian as a dimensionality detector; derives exact/replica-based spectra for a linear manifold model and interprets the empirical score as a random energy model whose condensation closes gaps, so high-variance subspaces are lost first to memorization.

Key results

• Gaps for higher-variance subspaces open earlier in diffusion time in the linear model, and no sharp gap appears for uniformly sampled variances.
• With small datasets no gap opens (full memorization); as training set size grows, the estimated manifold dimensionality increases on MNIST, with CelebA showing a clear transition and CIFAR10 a less evident one.
• Theory predicts a single-variance gap Delta(t)=sigma^2(1+alpha^{-1/2})^2/(t+sigma^2(1+alpha^{-1/2})^2), matching the replica-based spectrum qualitatively with some discrepancy at the spike near r=-1.

Measured effects reported in the paper

Surprising findings

• Subspaces with higher explained variance are lost earlier due to memorization (condensation), counter to intuition.
• Adding data points can sometimes lower the average estimated manifold dimension because different points have different dimensionality.

Limitations

• Theory relies on a linear data manifold model and small-time local linearization.
• Discrepancy between analytical and empirical spectra (missing spike at r=-1 in trained networks).
• Gaps become less visible in high-dimensional image datasets such as CIFAR10, with smooth rather than sharp transitions.

Experimental setup

Datasets: MNIST CIFAR10 CelebA synthetic linear manifold data (d=100, m=40)

Benchmarks: MNIST CIFAR10 CelebA

Metrics: singular value spectrum gaps estimated manifold dimensionality

Baselines: Stein-style intrinsic dimension estimator from prior work on diffusion manifolds

Builds on: denoising score matching manifold hypothesis random energy model

Closest papers by meaning

• Improving the Euclidean Diffusion Generation of Manifold Data by Mitigating Score Function Singularity (NeurIPS 2025 · similarity 0.91)
• Generative Modeling on Manifolds Through Mixture of Riemannian Diffusion Processes (ICML 2024 · similarity 0.90)
• What Secrets Do Your Manifolds Hold? Understanding the Local Geometry of Generative Models (ICLR 2025 · similarity 0.89)
• What Matters for Diffusion-Friendly Latent Manifold? Prior-Aligned Autoencoders for Latent Diffusion (NeurIPS 2026 · similarity 0.89)
• Diffusion Model's Generalization Can Be Characterized by Inductive Biases toward a Data-Dependent Ridge Manifold (NeurIPS 2026 · similarity 0.89)
• Hessian Geometry of Latent Space in Generative Models (ICML 2025 · similarity 0.88)