Research map / Image generation / Fast diffusion sampling

The two clocks and the innovation window: When and how generative models learn rules

NeurIPS 2026 · Image generation › Fast diffusion sampling · approach: Distributional control of diffusion · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Uses rule-valid synthetic binary tasks (group parity, exact-K, Latin squares, Sudoku) to show that diffusion (DiT) and autoregressive (GPT) models learn rules at a step tau_rule that depends on rule complexity, while memorization onset tau_mem scales with dataset size, opening an "innovation window" between them.

Problem

Generative models trained on finite data may memorize training samples rather than learn the latent rules defining the population distribution; the timing of rule learning versus memorization during training is not well characterized.

Approach

Defines two training clocks (rule-validity onset tau_rule and memorization onset tau_mem) and an innovation window between them; shows tau_rule is governed by rule complexity and model capacity while tau_mem scales nearly linearly with N, and dissects DiT score landscapes to explain the two transitions as attractor-basin changes.

Key results

• tau_mem fit to approximately 35 N^1.14 (R^2=0.94, n=34) for DiT-mini across all G; GPT-mini fit about 2.1 N^0.97 (R^2=0.94).
• tau_rule increases roughly an order of magnitude per unit increase in G: about 1e3 steps at G=2, 1e4 at G=3, 1e5 at G=4 for DiT-mini at N=4096.
• Exact-K and row-K rules are learned rapidly, comparable to parity with G in [2,3], showing that rule form (not globality alone) determines learning difficulty.

Surprising findings

• Combinatorial creativity is limited: for G<=12 most rule-valid generated groups are memorized training groups, so novel valid samples arise mainly by recombining memorized groups.
• GPT learns and memorizes faster than a matched-size DiT, yielding a shorter innovation window despite lower capacity under the same architecture.

Limitations

• Results are on synthetic rule-valid tasks with a fixed 6x6 binary image format (D=36), limiting transfer to natural data.
• Clock definitions depend on operational thresholds (rule accuracy >0.9, memorization >0.1).

Experimental setup

Datasets: group-parity synthetic dataset (D=36) exact-K synthetic dataset row-K synthetic dataset Latin square synthetic dataset Sudoku synthetic dataset

Benchmarks: group-parity rule accuracy exact-K rule accuracy row-K rule accuracy Latin square accuracy Sudoku accuracy

Metrics: sample-level rule accuracy group-level rule accuracy memorization ratio invalid (quantization) fraction denoising loss per noise scale cross-entropy

Baselines: GPT-mini

Builds on: EDM DiT GPT-2

Closest papers by meaning

• Stability and Diversity of Networked Self-Consuming Generative Ecosystems (NeurIPS 2026 · similarity 0.87)
• Self-Consuming Generative Models with Co-Evolving Human Preferences (NeurIPS 2026 · similarity 0.87)
• Emergence of Hidden Capabilities: Exploring Learning Dynamics in Concept Space (NeurIPS 2024 · similarity 0.87)
• Collapse or Thrive: Perils and Promises of Synthetic Data in a Self-Generating World (ICML 2025 · similarity 0.87)
• A Theoretical Perspective: How to Prevent Model Collapse in Self-consuming Training Loops (ICLR 2025 · similarity 0.87)
• How Compositional Generalization and Creativity Improve as Diffusion Models are Trained (ICML 2025 · similarity 0.86)