Research map / Image generation / Autoregressive and latent generation
D^2iT: Dynamic Diffusion Transformer for Accurate Image Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes D2iT, a two-stage dynamic-compression diffusion transformer that uses entropy-based multi-grained latents (DVAE) and predicts a grain map plus coarse/fine multi-grained noise to improve image generation quality and training efficiency over DiT.
Problem
Standard DiT applies fixed compression uniformly across image regions, ignoring varying information density; large compression hurts local realism while small compression raises cost and hurts global consistency.
Approach
A Dynamic VAE encodes regions at different downsampling factors (8x for detailed, 16x for smooth) chosen by local Shannon entropy; D2iT first predicts a grain map with a Dynamic Grain Transformer, then a Dynamic Content Transformer predicts multi-grained noise via coarse rough prediction and fine-grained RefineNet correction with windowed attention.
Key results
- D2iT-XL-G reaches FID-50K 1.73 on ImageNet 256x256 vs DiT-XL-G 2.27, with 57.1% of training steps and similar parameter count (687M vs 675M).
- D2iT-L achieves FID-10K 4.47 on unconditional FFHQ vs DiT-L 6.26 (about 28.6% improvement).
- Ablation on D2iT-B FFHQ: FID-10K drops from 34.67 (DiT-B) to 22.11 with the full dual-grained design.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use D2iT-L instead of DiT-L | FFHQ unconditional generation · FFHQ | FID-10K: 6.26 -> 4.47 | Helped |
| Use D2iT-XL-G instead of DiT-XL-G | ImageNet class-conditional generation · ImageNet 256×256 with classifier-free guidance | FID-50K: 2.27 -> 1.73 | Helped |
| Use D2iT-XL-G instead of MDTv2-G | ImageNet class-conditional generation · ImageNet 256×256 with classifier-free guidance | Inception Score: 314.73 -> 307.89 | Hurt |
| Add DVAE and Routing Module (ablation) | D2iT-B on FFHQ · FFHQ | FID-10K: 34.67 -> 29.10 | Helped |
| Add Dynamic grain loss (ablation) | D2iT-B on FFHQ · FFHQ | FID-10K: 29.10 -> 27.62 | Helped |
| Add RefineNet and learnable position embeddings (ablation) | D2iT-B on FFHQ · FFHQ | FID-10K: 27.62 -> 22.11 | Helped |
| Use 10 DiT layers and 2 RefineNet layers instead of 12 DiT layers (ablation) | D2iT-B on FFHQ · FFHQ | FID-10K: 25.15 -> 22.11 | Helped |
| Use 8 DiT layers and 4 RefineNet layers instead of 12 DiT layers (ablation) | D2iT-B on FFHQ · FFHQ | FID-10K: 25.15 -> 26.53 | Hurt |
| Add 2 RefineNet layers while keeping 12 DiT layers (ablation) | D2iT-B on FFHQ · FFHQ | FID-10K: 25.15 -> 20.99 | Helped |
| Add 4 RefineNet layers while keeping 12 DiT layers (ablation) | D2iT-B on FFHQ · FFHQ | FID-10K: 25.15 -> 19.96 | Helped |
| Replace the predicted grain map with a random grain map (ablation) | D2iT-L on FFHQ · FFHQ | FID-10K: 4.47 -> 15.93 | Hurt |
| Replace the predicted grain map with the ground-truth grain map (ablation) | D2iT-L on FFHQ · FFHQ | FID-10K: 4.47 -> 4.35 | Helped |
Surprising findings
- Using too many fine-grained codes for smooth regions hurts generation: FID rises from 23.64 to 29.15 when fine ratio goes from 0.7 to 1.0, despite better reconstruction rFID.
Limitations
- Only two granularities (8x and 16x) are explored in experiments.
- Evaluated only on FFHQ and ImageNet at 256x256 resolution.
Experimental setup
Datasets: ImageNet FFHQ
Benchmarks: ImageNet 256x256 FFHQ 256x256
Metrics: FID FID-10K FID-50K Inception Score Precision Recall rFID
Baselines: DiT-XL DiT-L MDT MDTv2 LDM-4 ADM VAR LlamaGen DiMR RDM UDM ImageBART
Builds on: Diffusion Transformer (DiT) DQ-VAE / Dynamic Quantization VAE latent diffusion models MDT
Closest papers by meaning
- Dynamic Diffusion Transformer (ICLR 2025 · similarity 0.93)
- Layer- and Timestep-Adaptive Differentiable Token Compression Ratios for Efficient Diffusion Transformers (CVPR 2025 · similarity 0.90)
- Faster Diffusion: Rethinking the Role of the Encoder for Diffusion Model Inference (NeurIPS 2024 · similarity 0.90)
- Accelerating Diffusion Transformer via Increment-Calibrated Caching with Channel-Aware Singular Value Decomposition (CVPR 2025 · similarity 0.89)
- FlexiDiT: Your Diffusion Transformer Can Easily Generate High-Quality Samples with Less Compute (CVPR 2025 · similarity 0.88)
- Autoregressive Distillation of Diffusion Transformers (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.