Research map / Architectures and efficiency / Attention and transformers
LazyDiT: Lazy Learning for the Acceleration of Diffusion Transformers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
LazyDiT adds lightweight linear layers before each MHSA and feedforward module of a frozen diffusion transformer to learn when to reuse cached outputs from the previous denoising step, skipping 20-50% of computation with better FID than DDIM at matched cost.
Problem
Diffusion transformers are slow at inference because every denoising step runs the full transformer; reducing per-step cost for transformer-based diffusion models has been less explored than for U-Net.
Approach
Observes that outputs of MHSA and feedforward modules at consecutive diffusion steps are highly similar, that this similarity is approximately linear in the current input (via Taylor expansion), and learns per-module linear laziness predictors that decide when to bypass computation and reuse the cached previous-step output.
Key results
- On DiT-XL/2 256x256, 20 steps with 50% lazy ratio gives FID 2.95 vs 3.53 for DDIM at 20 steps (IS 234.10 vs 222.87), and 10 steps with 50% lazy gives FID 6.75 vs 12.05 for DDIM.
- On Large-DiT-7B 256x256, 50 steps with 30% lazy ratio gives FID 2.13 vs 2.16 for DDIM at 50 steps, with lower cost.
- On a Snapdragon 8 Gen 3 phone, 10 steps with 30% lazy ratio gives IS 136.81 at 3.57 s versus DDIM 7 steps at 91.67 IS and 3.54 s.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use LazyDiT with 20% lazy ratio instead of DDIM at 50 steps | DiT-XL/2 · ImageNet 256×256 | FID: 2.34 -> 2.37 | No clear effect |
| Use LazyDiT with 50% lazy ratio instead of DDIM at 25 steps | DiT-XL/2 · ImageNet 256×256 | FID: 2.95 -> 2.70 | Helped |
| Use LazyDiT with 50% lazy ratio instead of DDIM at 25 steps | DiT-XL/2 · ImageNet 512×512 | FID: 4.26 -> 3.94 | Helped |
| Use LazyDiT with 50% lazy ratio instead of DDIM at 25 steps | Large-DiT-3B · ImageNet 256×256 | FID: 2.75 -> 2.42 | Helped |
| Use LazyDiT with 50% lazy ratio instead of DDIM at 25 steps | Large-DiT-7B · ImageNet 256×256 | FID: 2.76 -> 2.53 | Helped |
| Use LazyDiT with 50% lazy ratio instead of DDIM at 25 steps | DiT-XL/2 · ImageNet 256×256, mobile | IS: 230.95 -> 237.03 | Helped |
| Use LazyDiT with 50% lazy ratio instead of DDIM at 25 steps | DiT-XL/2 · ImageNet 512×512, mobile | IS: 192.71 -> 200.93 | Helped |
| Apply laziness to MHSA only rather than to MHSA and Feedforward together (ablation) | LazyDiT on DiT-XL/2 · ImageNet 256×256, DDIM 20 steps | generation performance | Hurt |
| Apply laziness to Feedforward only rather than to MHSA and Feedforward together (ablation) | LazyDiT on DiT-XL/2 · ImageNet 256×256, DDIM 20 steps | generation performance | Hurt |
| Use the same lazy ratio for MHSA and Feedforward (ablation) | LazyDiT on DiT-XL/2 · ImageNet 256×256, DDIM 20 steps | generation performance | Helped |
| Vary one module's lazy ratio while fixing the other at 30% for MHSA or 20% for Feedforward (ablation) | LazyDiT on DiT-XL/2 · ImageNet 256×256, DDIM 20 steps | generation performance | Mixed |
Surprising findings
- Removing whole layers degrades performance significantly, suggesting redundancy lies across sampling steps rather than in architecture depth.
- Learned laziness differs by layer: MHSA laziness decreases with depth while feedforward laziness increases.
Limitations
- Additional computation overhead from the lazy learning layers.
- Gains are reported mainly on class-conditional ImageNet DiT models; generalization to other tasks is not shown.
- Theoretical analysis relies on informal assumptions (e.g., small eta, Lipschitz constants) that are not verified empirically in full.
Experimental setup
Datasets: ImageNet
Benchmarks: ImageNet
Metrics: FID sFID IS Precision Recall TMACs latency
Baselines: DDIM Learn2Cache
Builds on: DiT DDIM Learn2Cache classifier-free guidance
Closest papers by meaning
- TinyFusion: Diffusion Transformers Learned Shallow (CVPR 2025 · similarity 0.90)
- SparseDiT: Token Sparsification for Efficient Diffusion Transformer (NeurIPS 2025 · similarity 0.89)
- BlockDance: Reuse Structurally Similar Spatio-Temporal Features to Accelerate Diffusion Transformers (CVPR 2025 · similarity 0.89)
- Learning-to-Cache: Accelerating Diffusion Transformer via Layer Caching (NeurIPS 2024 · similarity 0.89)
- FasterDiT: Towards Faster Diffusion Transformers Training without Architecture Modification (NeurIPS 2024 · similarity 0.88)
- HarmoniCa: Harmonizing Training and Inference for Better Feature Caching in Diffusion Transformer Acceleration (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.