Research map / Image generation / Autoregressive and latent generation
GenTron: Diffusion Transformers for Image and Video Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Explores transformer-based (DiT-style) latent diffusion for text-to-image and text-to-video generation, scaling to 3B parameters and adding motion-free guidance for video.
Problem
Visual diffusion generation relies on CNN U-Nets; DiT was limited to class conditioning and small scale, and text-to-video diffusion suffers from degraded per-frame quality after fine-tuning.
Approach
Adapts DiT from class to free-form text conditioning (comparing adaLN-Zero vs cross-attention and CLIP/T5 encoders), scales the transformer to over 3B parameters, and extends it to video by inserting a temporal self-attention layer per block trained with motion-free guidance (identity attention mask with classifier-free-style guidance on motion) plus joint image-video training.
Key results
- Human study vs SDXL on 100 PartiPrompts images (3000 votes): 51.1% win rate in visual quality (19.8% draw) and 42.3% win rate in text faithfulness (42.9% draw).
- GenTron-CLIP-T5XXL-G/2 reaches 49.99 mean on T2I-CompBench, above PixArt-α (48.15) and LDM XL (44.41), with color binding 76.74.
- GenTron-G/2 (3.1B) reaches CLIP-Score 0.335 vs 0.329 for SDXL, but FID-30K of 14.53 is worse than SD v1.4 (12.94) and SDXL (17.82 is worse still).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace adaLN-zero conditioning with cross-attention (ablation) | GenTron-XL/2 with CLIP-L · T2I-CompBench | mean score: 34.32 -> 47.84 | Helped |
| use T5-XXL instead of CLIP-L text embeddings (ablation) | GenTron-XL/2 with cross-attention · T2I-CompBench | mean score: 47.84 -> 48.93 | Helped |
| combine CLIP-L and T5-XXL text embeddings (ablation) | GenTron-XL/2 with cross-attention · T2I-CompBench | mean score: 48.93 -> 49.13 | Helped |
| scale from GenTron-XL/2 to GenTron-G/2 (ablation) | GenTron with CLIP-T5XXL and cross-attention · T2I-CompBench | mean score: 49.13 -> 49.99 | Helped |
| scale from GenTron-XL/2 to GenTron-G/2 (ablation) | GenTron with CLIP-T5XXL and cross-attention · T2I generation | CLIP-Score: 0.326 -> 0.335 | Helped |
| scale from GenTron-XL/2 to GenTron-G/2 (ablation) | GenTron with CLIP-T5XXL and cross-attention · T2I generation | FID-30K: 14.21 -> 14.53 | Hurt |
| add motion-free guidance (ablation) | GenTron-T2V · text-to-video generation | visual quality | Helped |
| use GenTron instead of SDXL | T2I generation · human evaluation, visual quality | win rate: 51.1% win rate (19.8% draw rate) | Helped |
| use GenTron instead of SDXL | T2I generation · human evaluation, text faithfulness | win rate: 42.3% win rate (42.9% draw rate) | Helped |
Surprising findings
- adaLN-Zero, the DiT default for class conditioning, underperforms cross-attention on free-form text conditioning across all T2I-CompBench metrics.
- GenTron-G/2 has worse FID-30K (14.53) than SD v1.4 (12.94), while winning on CLIP-Score and T2I-CompBench; the authors argue FID misaligns with human preference.
Limitations
- Training relies on internal, non-public datasets, limiting reproducibility.
- Video quality still lags per-frame T2I quality; the motion-free guidance benefit is shown qualitatively only (Figure 8) without quantitative video metrics.
- FID does not favor GenTron over SD v1.4 and SDXL, and the human study is limited to 100 PartiPrompts prompts.
Experimental setup
Datasets: internal image-text dataset (~550M pairs) internal video dataset (~34M videos) WebVid-10M (cited) PartiPrompts (evaluation prompts) T2I-CompBench
Benchmarks: T2I-CompBench FID-30K CLIP-Score on COCO PartiPrompts human study
Metrics: win rate draw rate T2I-CompBench attribute binding/object relationship scores FID-30K CLIP-Score
Baselines: SDXL SD v1.4 LDM v1.4 LDM v2 PixArt-α DALL·E 2 Imagen Parti GigaGAN MUSE Attn-Exct v2 GORS
Builds on: DiT latent diffusion models classifier-free guidance adaLN-Zero WebVid-10M
Closest papers by meaning
- Scaling Diffusion Transformers Efficiently via $\mu$P (NeurIPS 2025 · similarity 0.88)
- Towards Precise Scaling Laws for Video Diffusion Transformers (CVPR 2025 · similarity 0.87)
- ViDiT-Q: Efficient and Accurate Quantization of Diffusion Transformers for Image and Video Generation (ICLR 2025 · similarity 0.87)
- Representation Alignment for Generation: Training Diffusion Transformers Is Easier Than You Think (ICLR 2025 · similarity 0.87)
- ACE: All-round Creator and Editor Following Instructions via Diffusion Transformer (ICLR 2025 · similarity 0.87)
- From Slow Bidirectional to Fast Autoregressive Video Diffusion Models (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.