Research map / Image generation / Fast diffusion sampling
LiT: Delving into a Simple Linear Diffusion Transformer for Image Generation
Jiahao Wang, Ning Kang, Lewei Yao, Mengzhao Chen, Chengyue Wu, Songyang Zhang, Shuchen Xue, Yong Liu, Taiqiang Wu, Xihui Liu, Kaipeng Zhang, Shifeng Zhang and 3 more
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add depth-wise convolution on value (ablation) | ReLU linear attention DiT-S/2 · ImageNet 256 DiT-S/2 | FID: 88.46 -> 63.66 | Improved |
| add depth-wise convolution on value (ablation) | ReLU linear attention DiT-B/2 · ImageNet 256 DiT-B/2 | FID: 56.92 -> 42.11 | Improved |
| add focused function on top of DWC (ablation) | linear DiT-S/2 · ImageNet 256 DiT-S/2 | FID: 63.66 -> 63.05 | No clear effect |
| replace ReLU kernel with GELU (ablation) | focused linear DiT-S/2 · ImageNet 256 DiT-S/2 | FID: 63.05 -> 70.83 | Hurt |
| use few heads (2) instead of default 6 (ablation) | linear DiT-S/2 · ImageNet 256 DiT-S/2 | FID: 63.66 -> 63.24 | Improved |
| use 96 heads instead of 2 (ablation) | linear DiT-S/2 · ImageNet 256 DiT-S/2 | FID: 63.24 -> 116.00 | Hurt |
| use 2 heads instead of 16 (ablation) | linear DiT-XL/2 · ImageNet 256 DiT-XL/2 | FID: 21.69 -> 20.66 | Improved |
| initialize from DiT pretrained 200K steps (vs random) (ablation) | linear DiT-S/2 · ImageNet 256 DiT-S/2 | FID: 63.24 -> 57.84 | Improved |
| initialize from DiT pretrained 800K vs 200K steps (ablation) | linear DiT-S/2 · ImageNet 256 DiT-S/2 | FID: 57.84 -> 53.83 | Improved |
| also load Q,K,V attention projections from DiT (ablation) | linear DiT-S/2 · ImageNet 256 DiT-S/2 | FID: 54.80 -> 55.29 | Hurt |
| add noise distillation from larger DiT-XL/2 teacher (lambda1=0.5) (ablation) | linear DiT-S/2 · ImageNet 256 DiT-S/2 | FID: 53.83 -> 51.13 | Improved |
| add variance distillation lambda2=0.05 to noise distillation (ablation) | linear DiT-S/2 · ImageNet 256 DiT-S/2 | FID: 51.13 -> 50.79 | Improved |
Closest papers by meaning
- CLEAR: Conv-Like Linearization Revs Pre-Trained Diffusion Transformers Up (NeurIPS 2025 · similarity 0.91)
- DiCo: Revitalizing ConvNets for Scalable and Efficient Diffusion Modeling (NeurIPS 2025 · similarity 0.90)
- SD-DiT: Unleashing the Power of Self-supervised Discrimination in Diffusion Transformer (CVPR 2024 · similarity 0.89)
- Layer- and Timestep-Adaptive Differentiable Token Compression Ratios for Efficient Diffusion Transformers (CVPR 2025 · similarity 0.89)
- Scaling Diffusion Transformers Efficiently via $\mu$P (NeurIPS 2025 · similarity 0.89)
- PixArt-$\alpha$: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesis (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
