Research map / Image generation / Text-to-image diffusion
On improved Conditioning Mechanisms and Pre-training Strategies for Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Systematic re-implementation and study of five diffusion transformer/UNet architectures, proposing disentangled control conditioning, noisy-replicate text padding, and resolution-transfer tricks that set new FID results on ImageNet-1k and CC12M.
Problem
Large-scale latent diffusion training recipes are closed and hard to compare, and there is little study of how to condition on semantic and control metadata and how to transfer pre-trained weights across datasets and resolutions efficiently.
Approach
Decouple semantic conditioning (class/text via attention) from control metadata (crop, flip, size) with a cosine-scheduled low-level control embedding and a separate control guidance term; transfer low-resolution pre-trained weights by resampling sinusoidal positional grids, rescaling the noise schedule for higher resolution, and using more aggressive local cropping in pre-training.
Key results
- On ImageNet-1k at 512 resolution, the proposed mmDiT-XL/2 variant reaches FID 2.76, improving the prior DiT result of 3.04.
- On CC12M at 512 resolution, FID improves from 11.24 (train from scratch) to 6.27 with low-res pretraining, positional resampling and noise schedule rescaling, with CLIP COCO rising from 24.23 to 25.91.
- Replicate text padding with small noise (beta=0.02) improves CC12M@256 FID from 7.19 to 6.79 and CLIP from 26.25 to 26.60.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace uniform control weighting with cosine weighting of low-level controls (ablation) | DiT-XL/2 with control conditioning · ImageNet-1k 256 | LPIPS: 0.33 vs 0.04 | Helped |
| Use cosine low-level control weighting at inference (ablation) | DiT-XL/2 with control conditioning · ImageNet-1k 256, random size sampling | FID: 5.04 -> 3.80 | Helped |
| Add random-flip control conditioning (ablation) | mmDiT-XL/2 on CC12M · CC12M 256 | FID: 8.43 -> 8.40 | Helped |
| Replace zero text padding with noisy replicate padding (ablation) | mmDiT-XL/2 on CC12M · CC12M 256 | FID: 7.19 -> 6.79 | Helped |
| Replace zero text padding with noisy replicate padding (ablation) | mmDiT-XL/2 on CC12M · CC12M 256 | CLIP: 26.25 -> 26.60 | Helped |
| Pretrain on ImageNet-1k before ImageNet-22k finetuning (ablation) | mmDiT-XL/2 · ImageNet-22k 256 | FID: 5.80 -> 5.29 | Helped |
| Pretrain on ImageNet-1k before CC12M finetuning (ablation) | mmDiT-XL/2 · CC12M 256 | FID: 7.54 -> 7.27 | Helped |
| Pretrain mmDiT at 256 resolution before 512 finetuning (ablation) | mmDiT-XL/2 · ImageNet-1k 512 | FID: 3.98 -> 3.02 | Helped |
| Use local crops during low-resolution pretraining rather than global crops (ablation) | mmDiT-XL/2 · ImageNet-1k 512 adaptation | FID: almost 0.2 FID points | Helped |
| Use noisy replicate text padding with beta 0.02 (ablation) | mmDiT-XL/2 on CC12M · CC12M 256 | FID: 7.19 -> 6.79 | Helped |
| Use noisy replicate text padding with beta 0.02 (ablation) | mmDiT-XL/2 on CC12M · CC12M 256 | CLIP: 26.25 -> 26.60 | Helped |
| Use the proposed improved architecture and training recipe | mmDiT-XL/2 (SD3) · ImageNet-1k 512 | FID_train: 3.04 -> 2.76 | Helped |
Surprising findings
- Pre-training on low resolution with local (0.4-0.6) crops gives better 512-resolution FID than global crops, despite underperforming at low resolution.
- Standard DiT with cross-attention conditioning diverged in training, requiring RMSNorm on queries and keys for stability.
Limitations
- Gains from noisy replicate padding may diminish when models are trained only on long prompts.
- Flip conditioning only helps when data contains position-sensitive information; low-res local-crop pretraining may be unnecessary at very large data scales (>100M).
Experimental setup
Datasets: ImageNet-1k ImageNet-22k CC12M COCO
Benchmarks: ImageNet-1k 256 ImageNet-1k 512 CC12M 256 CC12M 512 COCO validation
Metrics: FID CLIPScore LPIPS
Baselines: DiT-XL/2 SDXL UNet mDT-v2-XL/2 PixArt-alpha-XL/2 mmDiT-XL/2 (SD3)
Builds on: DiT Stable Diffusion 3 (mmDiT) SDXL EDM PixArt-alpha
Closest papers by meaning
- Random Conditioning for Diffusion Model Compression with Distillation (CVPR 2025 · similarity 0.91)
- Würstchen: An Efficient Architecture for Large-Scale Text-to-Image Diffusion Models (ICLR 2024 · similarity 0.90)
- Learning Diffusion Models with Flexible Representation Guidance (NeurIPS 2025 · similarity 0.90)
- Plug-and-Play Diffusion Distillation (CVPR 2024 · similarity 0.90)
- Exploring the Role of Large Language Models in Prompt Encoding for Diffusion Models (NeurIPS 2024 · similarity 0.90)
- CommonCanvas: Open Diffusion Models Trained on Creative-Commons Images (CVPR 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.