Research map / Image generation / Autoregressive and latent generation
Probing the Latent Hierarchical Structure of Data via Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that forward-backward masked/Gaussian diffusion experiments reveal a correlation length and susceptibility that peak at the class phase transition of hierarchical (Random Hierarchy Model) data, and confirms this qualitatively in language (MDLM on WikiText2) and image (IDDPM on ImageNet with CLIP patch tokens) models.
Problem
Quantitative measurements of the hierarchical/compositional latent structure of real data are scarce, and latent variables are hard to access; it is unclear how changes in latent variables manifest in visible data.
Approach
Use forward-backward diffusion experiments as a probe: in the Random Hierarchy Model, token changes are correlated over a length scale that diverges at the class phase transition, and the dynamical susceptibility peaks there; this signature is then measured in real text and image diffusion models.
Key results
- In the RHM with v=32, m=8, s=2, L=9, the correlation length and susceptibility peak at the class phase transition (epsilon* ~0.74 for epsilon-process; t* ~0.3T for masking diffusion), matching mean-field theory.
- In MDLM on WikiText2, the correlation length reaches 7-8 tokens and susceptibility peaks at t* ~0.6T.
- For IDDPM on ImageNet with CLIP ViT-B/32 patch embeddings, susceptibility peaks at t* ~0.6-0.7T, a qualitative agreement with the theory.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use forward-backward diffusion on hierarchically structured RHM data near the class transition | Random Hierarchy Model ε-process · RHM ε-process | correlation length: maximal for ε* ≃ 0.74 | Helped |
| Use forward-backward diffusion on hierarchically structured RHM data near the class transition | Random Hierarchy Model masking diffusion · RHM masking diffusion | dynamical susceptibility: t* ≃ 0.3 T | Helped |
| Use MDLM forward-backward experiments at intermediate inversion time | MDLM on WikiText2 · WikiText2 | correlation length: 7 ÷ 8 tokens at t* ≈ 0.6 T | Helped |
| Use MDLM forward-backward experiments at the critical inversion time | MDLM on WikiText2 · WikiText2 | susceptibility: peaks at t* | Helped |
| Use forward-backward image diffusion near the class transition | Improved DDPM on ImageNet · ImageNet validation set | susceptibility: t* ≈ 0.6 ÷ 0.7 T | Helped |
| Use forward-backward diffusion on a Gaussian random field with spatial correlations | Gaussian random field · Gaussian random field | correlation length: maximal at the final time t=T | Mixed |
| Replace BP posterior sampling with backward diffusion using the BP score function | RHM masking diffusion · RHM masking diffusion | average correlation function | No clear effect |
| Replace BP posterior sampling with backward diffusion using the BP score function | RHM masking diffusion · RHM masking diffusion | dynamical susceptibility | No clear effect |
Surprising findings
- Spatial correlations alone (a Gaussian random field with power-law spectrum) do not produce a susceptibility peak at a finite inversion time; the correlation length grows monotonically up to t=T.
Limitations
- Image evaluation is only a qualitative agreement, with tokens defined via CLIP patch embeddings rather than pixels.
- The mapping between masking fraction and the epsilon noise level in the theory is not exact.
- The RHM is a simplified context-free grammar that does not capture context dependence of real languages.
Experimental setup
Datasets: WikiText2 ImageNet Random Hierarchy Model synthetic data
Benchmarks: WikiText2 ImageNet validation set
Metrics: normalized correlation function dynamical susceptibility correlation length
Baselines: Gaussian random field model
Builds on: Random Hierarchy Model masked diffusion language models Improved DDPM Belief Propagation
Closest papers by meaning
- Generalization Dynamics of Linear Diffusion Models (NeurIPS 2026 · similarity 0.89)
- How Compositional Generalization and Creativity Improve as Diffusion Models are Trained (ICML 2025 · similarity 0.89)
- Nested Diffusion Models Using Hierarchical Latent Priors (CVPR 2025 · similarity 0.89)
- Adapting to Unknown Low-Dimensional Structures in Score-Based Diffusion Models (NeurIPS 2024 · similarity 0.89)
- The Emergence of Reproducibility and Consistency in Diffusion Models (ICML 2024 · similarity 0.88)
- How Diffusion Models Learn to Factorize and Compose (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.