Research map / Image generation / Fast diffusion sampling

ToddlerDiffusion: Interactive Structured Image Generation with Cascaded Schrödinger Bridge

ICLR 2025 · Image generation › Fast diffusion sampling · approach: Diffusion theory and design space · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes ToddlerDiffusion, a cascaded modality-space diffusion pipeline (sketch → palette → RGB) that uses a Schrödinger Bridge formulation between stages instead of LDM concatenation, yielding better FID with smaller UNets and fewer denoising steps.

Problem

Standard latent diffusion models start each conditional stage from pure noise and rely on weak concatenation/cross-attention conditioning, which leads to slow training/sampling, large models, and poor controllability and editing.

Approach

Decompose RGB image generation into cascaded modality stages (sketch, palette, RGB) and connect them with a Schrödinger-Bridge-style forward process that blends the condition y into x_t, so each stage starts from a informative signal with high SNR; the model directly predicts x_0 and uses pseudo-GT conditions from edge detectors (PidiNet) and FastSAM-based palettes.

Key results

• On LSUN-Churches, ToddlerDiffusion reaches FID 6.19 vs. 7.30 for LDM at 250 epochs, and on CelebHQ FID 7.10 vs. 8.15 for LDM at 600 epochs.
• The UNet in the 3-stage pipeline is 104.8M parameters vs. 263M for LDM, with sampling at 2.27 fps vs. 1.21 fps.
• Training with 10 steps is reported to be sufficient with minimal impact on fidelity, and a 41x smaller sketch network achieves Sketch-FID 15.19 vs. 49 for LDM.

Measured effects reported in the paper

Surprising findings

• Models trained with fewer denoising steps (10) outperform those trained with 100 steps when later sampled with DDIM at 10 steps (FID ~15 vs ~60).
• The method yields almost the same performance regardless of training objective (epsilon vs. x0), contrary to prior findings that epsilon prediction is better.

Limitations

• Error accumulation across stages; generated sketches cause a domain gap (2nd-stage FID rises from 8.6 to 16.1) that needs condition truncation augmentation to mitigate.
• LDM results were reproduced by the authors due to replication difficulties; ImageNet experiments limited to ImageNet-100 due to limited resources.
• Failure cases yield unrealistic images when sketches are poor, especially in unconditional generation.

Experimental setup

Datasets: LSUN-Churches CelebHQ ImageNet-100 LAION-Art

Benchmarks: CelebHQ LSUN-Churches ImageNet-100

Metrics: FID KID precision recall

Baselines: LDM ControlNet SDEdit SEED-X

Builds on: DDPM latent diffusion models Schrödinger Bridge DDPM inversion DDIM

Closest papers by meaning

• DCTdiff: Intriguing Properties of Image Generative Modeling in the DCT Space (ICML 2025 · similarity 0.88)
• Finding Local Diffusion Schrodinger Bridge using Kolmogorov-Arnold Network (CVPR 2025 · similarity 0.87)
• Walking the Schrödinger Bridge: A Direct Trajectory for Text-to-3D Generation (NeurIPS 2025 · similarity 0.87)
• Improving Progressive Generation with Decomposable Flow Matching (NeurIPS 2025 · similarity 0.87)
• DiffuseHigh: Training-Free Progressive High-Resolution Image Synthesis Through Structure Guidance (AAAI 2025 · similarity 0.87)
• FAM Diffusion: Frequency and Attention Modulation for High-Resolution Image Generation with Stable Diffusion (CVPR 2025 · similarity 0.86)