Research map / Image generation / Fast diffusion sampling

Diffusion Models Encode the Intrinsic Dimension of Data Manifolds

ICML 2024 · Image generation › Fast diffusion sampling · approach: Diffusion theory and design space

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proves that diffusion model scores at small noise levels point toward the data manifold (approximating its normal bundle), and uses SVD of score vectors at a point to estimate the manifold's intrinsic dimension.

Problem

Estimating the intrinsic dimension of data manifolds from high-dimensional data, where existing statistical and normalizing-flow-based estimators struggle with high-ID or complex datasets.

Approach

At small noise levels the score of a noise-corrupted data distribution points toward the orthogonal projection onto the manifold, so the score vectors span the normal bundle; counting the near-vanishing singular values of a score matrix sampled around a point gives the local intrinsic dimension.

Key results

• On 50-sphere embedded in 100 dimensions, the method estimates 51 (true 50); on the spaghetti line it estimates 1 while PPCA gives 98.
• On Gaussian blobs k=100 the method estimates 98 with the other baselines (MLE, Local PCA, PPCA) underestimating or overestimating far more, e.g. PPCA gives 985.
• On MNIST, per-digit estimates range from 66 (digit 1) to 152 (digit 9), with ID-NF giving 182 overall and the method's MNIST estimate closely matching the auto-encoder reconstruction plateau.

Measured effects reported in the paper

Surprising findings

• Maximum-over-points estimate is used because pointwise estimates often underestimate the dimension, and MLE and Local PCA consistently underestimate MNIST ID.
• PPCA overestimates MNIST intrinsic dimension at 706.

Limitations

• Theory holds only in the limit of small noise and perfect score approximation; approximation and geometric errors can cause underestimation.
• Results are sensitive to non-uniform density on the manifold (alpha=0.5 underestimates) and to data concentrated only near rather than on the manifold.
• Choosing the cut-off point and the sampling point requires visual inspection or taking the maximum over many points.

Experimental setup

Datasets: synthetic Euclidean manifolds (k-spheres, spaghetti line, union of spheres) synthetic image manifolds (squares, Gaussian blobs) MNIST

Benchmarks: synthetic k-spheres spaghetti line squares image manifold Gaussian blobs image manifold MNIST

Metrics: estimated intrinsic dimension singular value spectrum autoencoder reconstruction error (MNIST)

Baselines: MLE (m=5) MLE (m=20) Local PCA PPCA ID-NF

Builds on: score-based SDE likelihood-weighted denoising score matching maximum likelihood ID estimator (Levina and Bickel) ID-NF

Closest papers by meaning

• Adapting to Unknown Low-Dimensional Structures in Score-Based Diffusion Models (NeurIPS 2024 · similarity 0.90)
• CW Complex Hypothesis for Image Data (ICML 2024 · similarity 0.90)
• Improving the Euclidean Diffusion Generation of Manifold Data by Mitigating Score Function Singularity (NeurIPS 2025 · similarity 0.89)
• Diffusion Models and the Manifold Hypothesis: Log-Domain Smoothing is Geometry Adaptive (NeurIPS 2025 · similarity 0.89)
• Revisiting Diffusion Model Predictions Through Dimensionality (NeurIPS 2026 · similarity 0.89)
• Diffusion Model's Generalization Can Be Characterized by Inductive Biases toward a Data-Dependent Ridge Manifold (NeurIPS 2026 · similarity 0.89)