Research map / Image generation / Fast diffusion sampling

FlexiDiT: Your Diffusion Transformer Can Easily Generate High-Quality Samples with Less Compute

CVPR 2025 · Image generation › Fast diffusion sampling · approach: Training-free diffusion acceleration · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Converts pre-trained Diffusion Transformers into FlexiDiTs that use different patch sizes per denoising step (weak patch for early steps), cutting FLOPs by over 40% for image and up to 75% for video generation without quality loss.

Problem

Diffusion Transformers spend a fixed, large amount of compute on every denoising step, although early steps mainly determine low-frequency structure and can tolerate less compute.

Approach

Make a pre-trained DiT flexible by adding new patch-size-specific embedding/de-embedding layers (initialized via pseudo-inverse of bilinear interpolation) so one model can tokenize the same image at a larger, cheaper patch size, then schedule weak (large-patch) steps early and powerful steps late, with guidance computed from the weak model's conditional prediction.

Key results

• On class-conditioned ImageNet DiT-XL/2, FlexiDiT with only powerful mode matches FID-50k 2.25 vs 2.27 for the pre-trained model, and reduces FLOPs by over 40% without quality drop.
• On text-to-image, the flexible models match baseline FID/CLIP/VQAScore while using less than 60% of original compute for T2I Transf. (Emu saves about 31% compute at equal quality).
• On text-to-video (Video DiT, 4.9B), up to 75% compute savings with no significant drop in VBench performance.

Measured effects reported in the paper

Surprising findings

• Using the weak (larger patch) model for the guidance signal can yield better generation quality than conventional unconditional guidance.
• Starting with the powerful model and switching to the weak model gives worse results than the proposed weak-first scheduler.

Limitations

• Savings come with a minor FID/quality drop when too many steps use the weak model.
• LoRA variant adds up to 5% parameters or small FLOPs overhead if unmerged, or extra memory if merged.
• Restricted to patch sizes larger than the pre-trained one; iterative use of multiple weak modes left for future work.

Experimental setup

Datasets: ImageNet MS COCO

Benchmarks: ImageNet FID-50k MS COCO FID and CLIP score DrawBench TIFA160 Pick-a-Pic Winoground VBench

Metrics: FID CLIP score VQAScore VBench score FLOPs latency human preference win/tie/lose

Baselines: DiT-XL/2 PixArt-alpha T2I Transf. Emu Video DiT training-free video acceleration methods

Builds on: DiT PixArt-alpha Emu LoRA FlexiViT

Closest papers by meaning

• Layer- and Timestep-Adaptive Differentiable Token Compression Ratios for Efficient Diffusion Transformers (CVPR 2025 · similarity 0.90)
• Remix-DiT: Mixing Diffusion Transformers for Multi-Expert Denoising (NeurIPS 2024 · similarity 0.90)
• Dynamic Diffusion Transformer (ICLR 2025 · similarity 0.89)
• Faster Diffusion: Rethinking the Role of the Encoder for Diffusion Model Inference (NeurIPS 2024 · similarity 0.89)
• HarmoniCa: Harmonizing Training and Inference for Better Feature Caching in Diffusion Transformer Acceleration (ICML 2025 · similarity 0.89)
• SparseDiT: Token Sparsification for Efficient Diffusion Transformer (NeurIPS 2025 · similarity 0.89)