Research map / Image generation / Autoregressive and latent generation

EDT: An Efficient Diffusion Transformer Framework Inspired by Human-like Sketching

NeurIPS 2024 · Image generation › Autoregressive and latent generation · approach: High-resolution diffusion synthesis · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes EDT, a lightweight transformer diffusion model with down/up-sampling, a training-free Attention Modulation Matrix that converts global to local attention in alternating decoder blocks, and a token-masking training strategy, reaching lower FID than MDTv2 at lower cost on ImageNet.

Problem

Transformer-based diffusion probabilistic models achieve strong image synthesis but have high training and inference computational cost, and few works address their efficiency.

Approach

A U-Net-like diffusion transformer with token down-sampling (reducing tokens 4x, dimension 1.25x) to cut FLOPs; a training-free Attention Modulation Matrix based on a cosine of token Euclidean distance, applied by Hadamard product to attention scores in alternating up-sampling blocks to mimic coarse-to-fine human sketching; and a masking strategy that feeds full tokens to shallow blocks and masks

Key results

• EDT-XL achieves FID 7.52 vs 7.70 for MDTv2-XL and 19.47 for DiT-XL without classifier-free guidance on ImageNet 256x256, with inference speed 0.51 step/s vs 0.23 for MDTv2-XL.
• Adding AMM to pre-trained DiT-XL reduces FID from 18.48 to 14.73; to MDTv2-S from 39.02 to 31.89.
• EDT-S reaches FID 34.27 at 5.50 inference steps/s versus MDTv2-S at 39.50 FID and 2.40 steps/s.

Measured effects reported in the paper

Surprising findings

• Applying AMM during training hurts performance (FID 52.1 vs 38.7 when added only at inference), so AMM works best as a training-free plug-in to pre-trained models.
• The MDT-style masking strategy gives only a small gain (FID 49.6) over no masking (50.9), while the EDT masking strategy gives 46.9.

Limitations

• The best AMM arrangement and hyperparameters vary across models and require testing.
• Some EDT results are without classifier-free guidance and the 512x512 comparison covers only small models.

Experimental setup

Datasets: ImageNet

Benchmarks: ImageNet 256x256 ImageNet 512x512

Metrics: FID sFID IS Precision Recall training speed (iter/s) inference speed (step/s) GFLOPs memory

Baselines: DiT MDTv2 SD-DiT ADM LDM-4 U-ViT

Builds on: DiT MDTv2 LDM ADM

Closest papers by meaning

• Exploring Diffusion Transformer Designs via Grafting (NeurIPS 2025 · similarity 0.89)
• Expert Race: A Flexible Routing Strategy for Scaling Diffusion Transformer with Mixture of Experts (ICML 2025 · similarity 0.88)
• SD-DiT: Unleashing the Power of Self-supervised Discrimination in Diffusion Transformer (CVPR 2024 · similarity 0.88)
• EC-DIT: Scaling Diffusion Transformers with Adaptive Expert-Choice Routing (ICLR 2025 · similarity 0.88)
• Layer- and Timestep-Adaptive Differentiable Token Compression Ratios for Efficient Diffusion Transformers (CVPR 2025 · similarity 0.88)
• Faster Diffusion: Rethinking the Role of the Encoder for Diffusion Model Inference (NeurIPS 2024 · similarity 0.88)