Research map / Image generation / Fast diffusion sampling

Swing-by Dynamics in Concept Learning and Compositional Generalization

ICLR 2025 · Image generation › Fast diffusion sampling · approach: Diffusion models as perception tools · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes a structured identity mapping (SIM) task on Gaussian mixtures as a theoretical abstraction of diffusion-model concept learning, analyzes linear/symmetric 2-layer dynamics to explain swing-by (transient memorization) and multiple-descent test loss, and validates with a text-conditioned diffusion model.

Problem

Theoretically explaining the empirically observed learning dynamics of compositional (OOD) generalization in text-conditioned diffusion models, such as sequential concept learning order and non-monotonic test loss.

Approach

Abstracts the concept space into a structured identity mapping regression task on a Gaussian mixture with structured centroids, and derives analytically that a symmetric 2-layer linear network's Jacobian passes through growth and suppression stages of major and minor entries, producing a transient swing toward the strongest training cluster (swing-by) and multiple descents in OOD loss.

Key results

• Theory for a one-layer linear model gives analytic solution where generalization order and terminal slowdown follow exp(-a_k t) with a_k=(s*sigma_k^2+mu_k^2)/s, but cannot reproduce non-monotonic swing-by.
• A symmetric 2-layer linear model reproduces swing-by: OOD loss shows double (or multiple, for s>2) descent tied to alternating growth and suppression of major and minor Jacobian entries.
• In text-conditioned diffusion experiments, concept-signal strength controls generalization order, a transient bend toward the stronger-signal concept appears with double-descent-like test MSE, and concept traversal speed decays exponentially.

Measured effects reported in the paper

Surprising findings

• Non-monotonic OOD test loss arises from a distributional mechanism (fitting the training distribution) rather than in-distribution noise or overparameterization, distinct from standard epoch-wise double descent.
• Low input dimension yields more pronounced swing-by, while high-dimensional settings show only slowed descent without non-monotonicity.

Limitations

• Theory relies on linear models, small-initialization and step-size assumptions, and a significant signal strength difference assumption.
• Diffusion validation is limited to a small synthetic setup with two concepts (size and color).

Experimental setup

Datasets: Synthetic Gaussian mixture SIM data Synthetic circle images with size and color concepts (setup from prior work)

Benchmarks: Out-of-distribution test point on SIM OOD test pair (blue, small) on concept-space diffusion task

Metrics: test loss (MSE) concept-space output trajectory concept traversal speed per step

Baselines: One-layer linear model theory Diagonally initialized linear networks

Builds on: Concept space framework for compositional generalization in diffusion models Epoch-wise double descent Deep linear network dynamics (Saxe et al.)

Closest papers by meaning

• How Compositional Generalization and Creativity Improve as Diffusion Models are Trained (ICML 2025 · similarity 0.90)
• Why Diffusion Models Don’t Memorize: The Role of Implicit Dynamical Regularization in Training (NeurIPS 2025 · similarity 0.89)
• Emergence of Hidden Capabilities: Exploring Learning Dynamics in Concept Space (NeurIPS 2024 · similarity 0.88)
• When does compositional structure yield compositional generalization? A kernel theory. (ICLR 2025 · similarity 0.88)
• Scaling can lead to compositional generalization (NeurIPS 2025 · similarity 0.87)
• Generalization Dynamics of Linear Diffusion Models (NeurIPS 2026 · similarity 0.87)