Research map / Image generation / Text-to-image diffusion
PixArt-$\alpha$: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesis
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Trains PixArt-α, a 0.6B DiT-based text-to-image diffusion model with cross-attention and adaLN-single, in a three-stage decomposed pipeline with LLaVA-recaptioned data, at about 753 A100 GPU days (~$28,400).
Problem
Text-to-image diffusion models require enormous training compute (millions of GPU hours, high cost and CO2 emissions), which blocks research and startups from building high-quality T2I models.
Approach
Decompose T2I training into pixel-dependency learning (initialized from an ImageNet class-conditional DiT via re-parameterization), text-image alignment on dense LLaVA-recaptioned data, and aesthetic fine-tuning; extend DiT with cross-attention for text and a shared adaLN-single modulation to cut parameters.
Key results
- Zero-shot COCO FID-30K of 7.32 with 753 A100 GPU days and 25M images, versus SDv1.5 at 6,250 A100 GPU days and 2B images.
- Best T2I-CompBench scores on 5 of 6 metrics, e.g., color 0.6886 and complex 0.4117.
- User study with 50 participants: 7.2% better image quality and 42.4% better alignment than SDv2.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use PixArt-α instead of RAPHAEL | T2I generation · MSCOCO zero-shot | FID-30K: 6.61 -> 7.32 | Hurt |
| use PixArt-α instead of the strongest reported baseline | T2I generation · T2I-CompBench color | Color: 0.6603 -> 0.6886 | Helped |
| use PixArt-α instead of the strongest reported baseline | T2I generation · T2I-CompBench shape | Shape: 0.5464 -> 0.5582 | Helped |
| use PixArt-α instead of the strongest reported baseline | T2I generation · T2I-CompBench texture | Texture: 0.6374 -> 0.7044 | Helped |
| use PixArt-α instead of the strongest reported baseline | T2I generation · T2I-CompBench spatial relationships | Spatial: 0.2032 -> 0.2082 | Helped |
| use PixArt-α instead of the strongest reported baseline | T2I generation · T2I-CompBench non-spatial relationships | Non-Spatial: 0.3193 -> 0.3179 | Hurt |
| use PixArt-α instead of the strongest reported baseline | T2I generation · T2I-CompBench complex compositions | Complex: 0.4091 -> 0.4117 | Helped |
| replace adaLN with adaLN-single (ablation) | PixArt-α · SAM test set | GPU memory: 29GB -> 23GB | Helped |
| replace adaLN with adaLN-single (ablation) | PixArt-α · SAM test set | parameters: 833M -> 611M | Helped |
| replace adaLN with adaLN-single (ablation) | PixArt-α · SAM test set | FID | Mixed |
| remove re-parameterization and train from scratch (ablation) | PixArt-α · SAM test set | visual quality | Hurt |
| extend adaLN-single training to 1500K iterations (ablation) | PixArt-α · SAM test set | FID | No clear effect |
Surprising findings
- The FID metric may not reflect visual quality; RAPHAEL has a lower FID but the authors argue FID is an inappropriate quality metric.
- Adding LLaVA-dense captions raised the valid-noun ratio on LAION from 8.5% to 13.3%, and the average nouns per image from 6.4 to 20.9.
Limitations
- Weak at precise counting, human limbs/hands, and text rendering.
- Training cost excludes VAE training, T5 feature extraction, and LLaVA auto-labeling time.
- FID is argued to be unreliable, so quality claims depend largely on a 50-person user study.
Experimental setup
Datasets: ImageNet SAM LAION JourneyDB internal 10M high-quality dataset MSCOCO
Benchmarks: MSCOCO FID-30K (zero-shot) T2I-CompBench user study on 300 prompts
Metrics: FID CLIP score T2I-CompBench attribute binding, object relationship, comple human preference rate GPU days GPU memory
Baselines: DALL·E 2 Stable Diffusion v1.5 Imagen RAPHAEL GigaGAN SDXL DeepFloyd Stable Diffusion v2 Midjourney
Builds on: DiT Latent Diffusion Model ControlNet DreamBooth DPM-Solver
Closest papers by meaning
- KOALA: Empirical Lessons Toward Memory-Efficient and Fast Diffusion Models for Text-to-Image Synthesis (NeurIPS 2024 · similarity 0.90)
- Exploring the Deep Fusion of Large Language Models and Diffusion Transformers for Text-to-Image Synthesis (CVPR 2025 · similarity 0.90)
- SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis (ICLR 2024 · similarity 0.89)
- Scaling Diffusion Transformers Efficiently via $\mu$P (NeurIPS 2025 · similarity 0.89)
- Stretching Each Dollar: Diffusion Training from Scratch on a Micro-Budget (CVPR 2025 · similarity 0.89)
- EC-DIT: Scaling Diffusion Transformers with Adaptive Expert-Choice Routing (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.