Research map / Image generation / Text-to-image diffusion
On the Scalability of Diffusion-based Text-to-Image Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Empirically studies scaling of diffusion-based text-to-image models by controlled comparisons of UNet and Transformer denoising backbones (0.4B–4B params) and training data (up to ~600M image-text pairs with synthetic captions), and fits power-law scaling functions for text-image alignment.
Problem
Scaling properties of diffusion-based text-to-image models are not well understood, and existing models differ in datasets and training settings, making fair comparison of denoising backbones and data scaling choices difficult.
Approach
Trains all backbone variants (SD2, DeepFloyd, SDXL UNets and scaled PixArt-alpha Transformers) under identical data, latent space, text encoder and training settings, identifies that transformer depth is more parameter-efficient than channel width for alignment, and shows that quality and caption density of training data matter more than raw size, then fits power-law scaling functions for TIFA ver
Key results
- SDXL-style UNet with transformer depth [0,4,4] (TD4_4) matches SDXL UNet TIFA with 45% fewer parameters and 28% less inference compute.
- Combining LensArt and SSTK with synthetic captions makes SDXL reach 0.82+ TIFA in 100K steps, 2.5x faster than SDXL trained on LensArt alone.
- Fitted power laws give S = 0.47C^0.02, S = 0.77N^0.11, S = 0.64D^0.03 for TIFA versus compute, model size and dataset size; TIFA correlates slightly better with FLOPs than parameters.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add randomly selected Top5 synthetic captions to LensArt training (ablation) | SD2 · LensArt, 250K steps | TIFA: 0.810 -> 0.835 | Helped |
| Add randomly selected Top5 synthetic captions to LensArt training (ablation) | SD2 · LensArt, 250K steps | ImageReward: 0.345 -> 0.524 | Helped |
| Add randomly selected Top5 synthetic captions to LensArt training (ablation) | SD2 · LensArt, 250K steps | FID: 17.9 -> 18.9 | Hurt |
| Add Top1 synthetic captions to LensArt training (ablation) | SD2 · LensArt, 250K steps | TIFA: 0.810 -> 0.833 | Helped |
| Add Top1 synthetic captions to LensArt training (ablation) | SD2 · LensArt, 250K steps | ImageReward: 0.345 -> 0.513 | Helped |
| Add Top1 synthetic captions to LensArt training (ablation) | SD2 · LensArt, 250K steps | FID: 17.9 -> 18.3 | Hurt |
| Use randomly selected Top5 rather than Top1 synthetic captions (ablation) | SD2 · LensArt, 250K steps | ImageReward: 0.513 -> 0.524 | Helped |
| Use randomly selected Top5 rather than Top1 synthetic captions (ablation) | SD2 · LensArt, 250K steps | CLIP: 0.271 -> 0.270 | Mixed |
| Increase UNet transformer depth rather than channel count (ablation) | SDXL UNet · TIFA evaluation during training | TIFA | Helped |
| Scale UNet channel count from 320 to 384 (ablation) | SDXL UNet · TIFA evaluation during training | TIFA | Helped |
| Combine LensArt and SSTK training data (ablation) | SD2 · TIFA and ImageReward during training | TIFA | Helped |
| Train on filtered LensArt instead of LensArt-raw (ablation) | SD2 · Generated images | aesthetic score: 5.07 -> 5.20 | Helped |
Surprising findings
- PixArt-alpha Transformer variants trained from scratch underperform SD2 UNet at equal steps, requiring more compute to match UNet performance.
- Increasing channel width is effective but compute-inefficient, whereas increasing transformer depth gives better alignment per parameter.
Limitations
- Experiments are run at 256x256 resolution; high-resolution behavior is only partly checked in appendix.
- Datasets LensArt and SSTK are internal and not released, limiting reproducibility.
- Transformer variants trained from scratch without ImageNet pretraining lag behind UNets.
Experimental setup
Datasets: LensArt SSTK MSCOCO-10K prompts TIFA prompts
Benchmarks: TIFA ImageReward MSCOCO-10K
Metrics: TIFA score ImageReward CLIP score FID HPSv2 aesthetic score training FLOPs
Baselines: SD2 UNet SDXL UNet DeepFloyd IF-XL UNet PixArt-alpha
Builds on: SDXL Stable Diffusion 2 DeepFloyd IF PixArt-alpha DiT
Closest papers by meaning
- SANA 1.5: Efficient Scaling of Training-Time and Inference-Time Compute in Linear Diffusion Transformer (ICML 2025 · similarity 0.91)
- EC-DIT: Scaling Diffusion Transformers with Adaptive Expert-Choice Routing (ICLR 2025 · similarity 0.90)
- One-Way Ticket: Time-Independent Unified Encoder for Distilling Text-to-Image Diffusion Models (CVPR 2025 · similarity 0.89)
- Improving Long-Text Alignment for Text-to-Image Diffusion Models (ICLR 2025 · similarity 0.89)
- Scale Where It Matters: Training-Free Localized Scaling for Diffusion Models (NeurIPS 2026 · similarity 0.89)
- Scaling Diffusion Transformers Efficiently via $\mu$P (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.