Research map / Image generation / Fast diffusion sampling
How Diffusion Models Learn to Factorize and Compose
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Controlled toy experiments on conditional DDPMs trained on 2D Gaussian bump/SOS images show they learn factorized but not fully continuous representations, compose well but interpolate poorly, need few compositional examples, and link manifold formation to percolation theory.
Problem
Whether and when diffusion models learn factorized representations of composable continuous features, and how this relates to compositional generalization and interpolation, which prior work on large models has not resolved.
Approach
Uses a highly reduced 2D Gaussian bump/SOS image setting with explicit (x,y) conditioning, analyzing both generated outputs and internal layer-4 representations via persistent homology, torus orthogonality/parallelism tests, and percolation theory to probe factorization and compositionality.
Key results
- Models learn factorized but not fully continuous representations: x-on-y orthogonality matches Clifford torus, while x-on-x and y-on-y statistics resemble orthogonal 3D-torus behavior rather than parallel Clifford structure.
- The 2D model composes but does not interpolate: high accuracy in vertical/horizontal test sections but low in the intersection, while the 2D+1D model generalizes across all test regions.
- For Gaussian bump + 1D stripe data, required dataset size to reach 60% accuracy scales linearly with N versus quadratically for bump-only data; subsampling bumps to 20% retains similar compositional performance.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add full-range 1D Gaussian stripes and some compositional examples to the training data | Conditional DDPM · Gaussian bump OOD test regions | accuracy: up to 70% | Helped |
| Train on 2D Gaussian SOSs plus full-range 1D stripes instead of 2D SOSs alone (ablation) | Conditional DDPM · Gaussian SOS held-out test regions | accuracy | Helped |
| Remove 1D stripes from the 2D+1D Gaussian SOS training data (ablation) | Conditional DDPM · Gaussian SOS held-out test regions | accuracy | Hurt |
| Remove compositional examples and train only on 1D Gaussian stripes (ablation) | Conditional DDPM · Gaussian SOS held-out test regions | accuracy | Hurt |
| Add a small set of partially overlapping 2D SOS examples along the image rim to 1D stripes (ablation) | Conditional DDPM · Gaussian SOS OOD regions | accuracy | Helped |
| Subsample 2D bump examples while retaining full-range 1D stripes (ablation) | 2D bumps + 1D stripes · Gaussian bump test regions | accuracy: 20% | No clear effect |
| Increase the held-out range width (ablation) | 2D Gaussian SOS model · Gaussian SOS OOD region | accuracy | Hurt |
| Replace Gaussian SOS training data with Gaussian bumps in the 2D+1D dataset (ablation) | Conditional DDPM · Interior and exterior compositional generalization | accuracy | Mixed |
| Use Gaussian SOS data instead of Gaussian bump data (ablation) | Conditional DDPM · Interior and exterior compositional generalization | accuracy | Helped |
| Decrease Gaussian bump width while holding the sampled grid points fixed (ablation) | Conditional DDPM · Gaussian bump datasets at varying data fractions | accuracy | Hurt |
| Increase the training fraction of very narrow σ=0.1 Gaussian bumps (ablation) | Conditional DDPM · σ=0.1 Gaussian bump dataset | accuracy: λ=0.8, 0.9 | Helped |
| Add isolated-factor examples plus a few compositional examples rather than training only on 2D bumps (ablation) | Conditional DDPM · OOD generalization across image sizes | data scaling: linear number of data in N; quadratic number of data in N | Helped |
Surprising findings
- Models learn hyper-factorized representations where different values of the same feature are nearly orthogonal rather than parallel, so the effective dimension exceeds the intrinsic 4D.
- Models reach compositional generalization from very few compositional examples (e.g., only rim examples), and the 1D model fails to compose despite learning factorized x and y.
Limitations
- Highly reduced toy setting with 2D Gaussian images; unclear transfer to natural images.
- Interpolation is limited; the generalization picture depends on specific held-out designs.
- Training variance is high in percolation experiments due to gradual rather than sharp transition.
Experimental setup
Datasets: 2D Gaussian bump images (32x32, synthetic) 2D Gaussian sum-of-stripes (SOS) images (synthetic) 1D Gaussian stripe images (synthetic)
Benchmarks: synthetic 2D Gaussian bump/SOS generation accuracy (held-out regions)
Metrics: accuracy of generated Gaussian center location participation ratio (effective dimension) persistent homology Wasserstein distance of orthogonality/parallelism statistics
Baselines: 1D model 2D model 2D+1D model
Builds on: DDPM UMAP Okawa et al. compositional generalization study
Closest papers by meaning
- How Compositional Generalization and Creativity Improve as Diffusion Models are Trained (ICML 2025 · similarity 0.91)
- Diffusion Classifiers Understand Compositionality, but Conditions Apply (NeurIPS 2025 · similarity 0.89)
- Compositional Discrete Latent Code for High Fidelity, Productive Diffusion Models (NeurIPS 2025 · similarity 0.89)
- What Matters for Diffusion-Friendly Latent Manifold? Prior-Aligned Autoencoders for Latent Diffusion (NeurIPS 2026 · similarity 0.89)
- Isometric Representation Learning for Disentangled Latent Space of Diffusion Models (ICML 2024 · similarity 0.89)
- Probing the Latent Hierarchical Structure of Data via Diffusion Models (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.