Research map / Image generation / Text-to-image diffusion
Towards Understanding the Working Mechanism of Text-to-Image Diffusion Model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Empirically and theoretically analyzes the denoising process of Stable Diffusion, finding shape is formed early via low-frequency signals and that the [EOS] token dominates text influence in that early stage, then uses this to skip text-conditioned evaluations late in sampling for ~25% speedup.
Problem
The working mechanism of latent diffusion text-to-image models, in particular how text prompt tokens influence generation across denoising steps, is poorly understood.
Approach
Shows that overall shape is determined in early denoising steps (low-frequency signals survive forward noising) and that the [EOS] token carries most of the text influence in that early stage; exploits this by dropping text-conditioned noise prediction after an early step a, giving training-free sampling acceleration.
Key results
- Skipping text guidance from step a=20 in DDIM on SD v1.5 saves about 27% latency while keeping FID near the baseline (13.8 vs 13.8).
- Switching the [EOS] token between prompts makes generated images align more with the prompt that supplied the [EOS] token than with the semantic-token prompt.
- Removing text guidance at large t (a from 30 to 50) quickly removes text influence on generated images, supporting that text info is conveyed early.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Remove text guidance for t < 20 | Stable Diffusion v1.5 with DDIM · MS-COCO test | Saved latency: +18.10% saved latency | Helped |
| Remove text guidance for t < 20 | Stable Diffusion v2.1 with DDIM · MS-COCO test | Text-CLIPScore: 31.413 -> 31.362 | No clear effect |
| Remove text guidance for t < 10 | Pixart-Alpha with DDIM · MS-COCO test | Text-CLIPScore: 28.157 -> 25.719 | Hurt |
| Remove text guidance for t < 10 | Pixart-Alpha with DPM-Solver · MS-COCO test | Text-CLIPScore: 30.733 -> 30.745 | No clear effect |
| Replace semantic tokens with zero vectors while preserving EOS (ablation) | Stable Diffusion v1.5 · PromptSet text-image alignment | CLIPScore (Text): 0.2407 -> 0.2999 | Helped |
| Replace EOS tokens with zero vectors while preserving semantic tokens (ablation) | Stable Diffusion v1.5 · PromptSet text-image alignment | CLIPScore (Text): 0.3110 -> 0.2407 | Hurt |
| Replace semantic tokens with random Gaussian noise while preserving EOS (ablation) | Stable Diffusion v1.5 · PromptSet text-image alignment | CLIPScore (Text): 0.2153 -> 0.3008 | Helped |
| Substitute EOS in Value only rather than Key only (ablation) | Stable Diffusion v1.5 · Switched-EOS prompts | CLIPScore (Text), target alignment: 0.1875 -> 0.2628 | Helped |
Surprising findings
- The [EOS] token, not the semantic tokens, dominates text influence on overall shape, and [SOS] receives most cross-attention weight despite carrying no semantic information.
- Random or zero vectors substituted for semantic tokens still yield semantically meaningful images when [EOS] is preserved.
Limitations
- Acceleration only holds for early-step removal; quality degrades sharply for large a (e.g. CLIPScore drops at a=40-50).
- Frequency analysis relies on a proposition for Gaussian noise and an approximate low-frequency threshold of 20%.
- Explanations of [EOS] dominance are partly speculative and attributed to token count and auto-regressive encoding.
Experimental setup
Datasets: PromptSet (1600 template prompts plus 1000 complex prompts) MS-COCO
Benchmarks: MS-COCO
Metrics: CLIPScore BLIP-VQA MiniGPT4-CoT L1-distance FID saved latency
Baselines: DDIM DPM-Solver
Builds on: Stable Diffusion classifier-free guidance DDIM DPM-Solver
Closest papers by meaning
- Schedule On the Fly: Diffusion Time Prediction for Faster and Better Image Generation (CVPR 2025 · similarity 0.89)
- InitNO: Boosting Text-to-Image Diffusion Models via Initial Noise Optimization (CVPR 2024 · similarity 0.88)
- Diffusion PID: Interpreting Diffusion via Partial Information Decomposition (NeurIPS 2024 · similarity 0.88)
- Diffusion in Diffusion: Cyclic One-Way Diffusion for Text-Vision-Conditioned Generation (ICLR 2024 · similarity 0.88)
- RealCompo: Balancing Realism and Compositionality Improves Text-to-Image Diffusion Models (NeurIPS 2024 · similarity 0.88)
- Denoising Autoregressive Transformers for Scalable Text-to-Image Generation (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.