Research map / Image generation / Text-to-image diffusion
SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Presents SDXL, a 2.6B-parameter latent diffusion UNet for text-to-image synthesis with size/crop micro-conditioning, multi-aspect training, an improved autoencoder, and a separate SDEdit-style refinement model.
Problem
Text-to-image latent diffusion models like Stable Diffusion 1.x/2.x have limited quality, produce cropped objects, waste data via minimum-resolution filtering, and are restricted to square outputs, while closed black-box generators remain stronger.
Approach
Scale the UNet ~3x with heterogeneous transformer blocks and a second text encoder (OpenCLIP ViT-bigG + CLIP ViT-L), add non-supervised conditioning on original size, crop coordinates, and target aspect-ratio bucket, and use a separate refinement LDM applied via noising-denoising.
Key results
- User study win rates: SDXL with refiner 48.44%, SDXL base 36.93%, SD 1.5 7.91%, SD 2.1 6.71%.
- UNet of 2.6B parameters with text encoders totaling 817M parameters, roughly 3x larger than previous SD UNet.
- Against Midjourney v5.1 on PartiPrompts, SDXL shows a slight preference in prompt adherence, while FID is worst among compared models despite better human ratings.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add the refinement stage to SDXL (ablation) | SDXL · User study | win rate: 36.93% -> 48.44% | Helped |
| use size conditioning rather than no size conditioning (ablation) | class-conditional LDM on ImageNet at 512² · ImageNet | IS | Helped |
| use size conditioning rather than no size conditioning (ablation) | class-conditional LDM on ImageNet at 512² · ImageNet | FID | Helped |
| discard training examples with an edge smaller than 512 pixels (ablation) | class-conditional LDM on ImageNet at 512² · ImageNet | IS | Hurt |
| discard training examples with an edge smaller than 512 pixels (ablation) | class-conditional LDM on ImageNet at 512² · ImageNet | FID | Hurt |
| add size conditioning (ablation) | class-conditional LDM on ImageNet at 512² · ImageNet | FID | Helped |
| add crop conditioning (ablation) | Stable Diffusion text-to-image model · Image generation | object cropping | Helped |
| add the refinement stage (ablation) | SDXL · Detailed backgrounds and human faces | visual sample quality | Helped |
| use SDXL rather than previous Stable Diffusion versions | SDXL · COCO zero-shot | FID | Hurt |
| use SDXL rather than previous Stable Diffusion versions | SDXL · COCO zero-shot | CLIP score | Helped |
| use transformer-based architectures such as UViT and DiT (ablation) | Stable Diffusion model development · Image generation | performance | No clear effect |
| use SDXL rather than Midjourney v5.1 | text-to-image generation · PartiPrompts | prompt adherence preference | Helped |
Surprising findings
- Classical metrics such as COCO zero-shot FID do not reflect human-preferred quality: SDXL has the worst FID of the compared models but only slightly better CLIP score.
- Discarding training images below 256 pixels would remove about 39% of the pretraining data, motivating size conditioning instead of filtering.
Limitations
- Two-stage pipeline with refiner increases memory and sampling cost.
- Struggles with human hands, concept bleeding, and long legible text rendering.
- Classical FID/CLIP metrics do not capture human-perceived quality improvements.
Experimental setup
Datasets: internal pretraining dataset ImageNet PartiPrompts (P2) COCO
Benchmarks: PartiPrompts (P2) COCO-10k zero-shot FID and CLIP ImageNet 512 IS/FID
Metrics: user preference win rate prompt adherence votes FID CLIP score IS
Baselines: Stable Diffusion 1.5 Stable Diffusion 2.1 Midjourney v5.1 DALL-E 3 DeepFloyd IF Midjourney
Builds on: Stable Diffusion latent diffusion models SDEdit classifier-free guidance offset-noise
Closest papers by meaning
- Any-Size-Diffusion: Toward Efficient Text-Driven Synthesis for Any-Size HD Images (AAAI 2024 · similarity 0.90)
- Diffusion-4K: Ultra-High-Resolution Image Synthesis with Latent Diffusion Models (CVPR 2025 · similarity 0.90)
- KOALA: Empirical Lessons Toward Memory-Efficient and Fast Diffusion Models for Text-to-Image Synthesis (NeurIPS 2024 · similarity 0.89)
- ScaleDiff: Higher-Resolution Image Synthesis via Efficient and Model-Agnostic Diffusion (NeurIPS 2025 · similarity 0.89)
- PixArt-$\alpha$: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesis (ICLR 2024 · similarity 0.89)
- FAM Diffusion: Frequency and Attention Modulation for High-Resolution Image Generation with Stable Diffusion (CVPR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.