Research map / Image generation / Text-to-image diffusion
Würstchen: An Efficient Architecture for Large-Scale Text-to-Image Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Würstchen, a three-stage latent diffusion architecture that uses a highly compressed (42:1) semantic latent to guide diffusion, achieving SD-competitive text-to-image quality at about 8x lower training cost.
Problem
Large-scale text-to-image diffusion models are computationally expensive to train and run; encoder-based latent diffusion models are limited by how much the autoencoder can compress images without degradation.
Approach
Decouple generation into three stages: a text-conditional diffusion model (Stage C) operating in a very compact semantic latent (16x24x24, 42:1 compression) from an EfficientNetV2-S Semantic Compressor, a latent diffusion model (Stage B) that reconstructs a 4x-compressed VQGAN latent conditioned on that semantic latent, and a VQGAN decoder (Stage A).
Key results
- Stage C training cost of 24,602 A100 GPU hours versus 200,000 for Stable Diffusion 2.1, described as about 8x cheaper.
- PickScore on Parti-prompts: Würstchen preferred over SD 1.4 at 82.1% and over SD 2.1 at 74.6%.
- Achieves FID 23.6 and IS 40.9 on COCO at 256 resolution in its comparison table, with a higher IS than SD 1.4/2.1 but higher FID than them (SD 2.1 FID 15.1).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the three-stage Würstchen architecture instead of Baseline LDM | 1B-parameter text-to-image generation · COCO-30K | PickScore preference: 96.5% | Helped |
| Use the three-stage Würstchen architecture instead of Baseline LDM | 1B-parameter text-to-image generation · Localized Narratives-COCO-5K | PickScore preference: 96.6% | Helped |
| Use the three-stage Würstchen architecture instead of Baseline LDM | 1B-parameter text-to-image generation · Parti-prompts | PickScore preference: 98.6% | Helped |
| Use Würstchen instead of SD 2.1 | text-to-image generation · COCO-30K | PickScore preference: 64.4% | Helped |
| Use Würstchen instead of SD 2.1 | text-to-image generation · Localized Narratives-COCO-5K | PickScore preference: 70.0% | Helped |
| Use Würstchen instead of SD 2.1 | text-to-image generation · Parti-prompts | PickScore preference: 74.6% | Helped |
| Use Würstchen instead of SD XL | text-to-image generation · COCO-30K | PickScore preference: 39.4% | Hurt |
| Use Würstchen instead of SD XL | text-to-image generation · Localized Narratives-COCO-5K | PickScore preference: 39.1% | Hurt |
| Use Würstchen instead of SD XL | text-to-image generation · Parti-prompts | PickScore preference: 39.0% | Hurt |
| Use Würstchen instead of Baseline LDM | COCO-30K text-to-image generation · COCO-30K | FID @256²: 43.5* -> 23.6* | Helped |
| Use Würstchen instead of Baseline LDM | COCO-30K text-to-image generation · COCO-30K | IS @299²: 20.1* -> 40.9* | Helped |
| Remove text conditioning from Stage B (ablation) | Würstchen · text-to-image generation | image quality | No clear effect |
Surprising findings
- Stage B's text conditioning appears to not improve image quality; Stage C is the primary driver of content, and Stage B mainly adds detail.
- Standard FID is highly sensitive to mild JPEG compression (FID rises to about 10 at 70% quality), so FID comparisons may be unreliable.
Limitations
- Higher FID than Stable Diffusion models despite higher IS and PickScore; authors attribute this to smoother outputs and metric sensitivity.
- SD XL is superior in image quality; comparison with it is acknowledged as unfair and was omitted from later experiments.
- Human study results on MS-COCO were inconclusive overall and depended on participant selection.
Experimental setup
Datasets: LAION-5B improved-aesthetic MS COCO validation Localized Narratives COCO Parti-prompts
Benchmarks: COCO-30K Localized Narratives-COCO-5K Parti-prompts
Metrics: PickScore preference rate FID Inception Score human preference win rate GPU hours estimated carbon emissions
Baselines: Stable Diffusion 1.4 Stable Diffusion 2.1 SD XL DF-GAN GALIP Baseline LDM CogView2 GLIDE Parti DALL-E LDM
Builds on: Latent Diffusion Models VQGAN DDPM classifier-free guidance DALL-E 2
Closest papers by meaning
- Dual Diffusion for Unified Image Generation and Understanding (CVPR 2025 · similarity 0.91)
- Exploring the Deep Fusion of Large Language Models and Diffusion Transformers for Text-to-Image Synthesis (CVPR 2025 · similarity 0.91)
- Noise Diffusion for Enhancing Semantic Faithfulness in Text-to-Image Synthesis (CVPR 2025 · similarity 0.91)
- Any-Size-Diffusion: Toward Efficient Text-Driven Synthesis for Any-Size HD Images (AAAI 2024 · similarity 0.90)
- On improved Conditioning Mechanisms and Pre-training Strategies for Diffusion Models (NeurIPS 2024 · similarity 0.90)
- Progressive Text-to-Image Diffusion with Soft Latent Direction (AAAI 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.