Research map / Architectures and efficiency / Attention and transformers
FasterDiT: Towards Faster Diffusion Transformers Training without Architecture Modification
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FasterDiT, which accelerates DiT training without architecture changes by analyzing SNR probability density functions to pick timestep sampling (logit-normal with std adjustment) and adding a velocity-direction cosine loss, reaching FID 2.30 at 1000k iterations versus DiT's 2.27 at 7000k.
Problem
Diffusion Transformers (DiT) converge slowly and are expensive to train; common SNR-modulating training strategies (noise schedules, loss weighting, timestep sampling) do not work consistently across data with different signal intensities.
Approach
Extends the SNR definition with a data-dependent scale (K(I) ∝ std²) and visualizes the PDF of training SNR to explain robustness trade-offs; from this, it tunes data std and logit-normal timestep sampling, and adds a cosine-similarity velocity-direction loss on top of MSE for velocity prediction.
Key results
- FasterDiT reaches FID 2.30 at 1000k iterations with cfg=1.5 on ImageNet 256, versus DiT's 2.27 at 7000k iterations (about 7x faster).
- On ImageNet 512, FasterDiT with DiT-B/2 improves FID-10k from 77.11 to 58.33 after 200k iterations at batch 128.
- Adding the velocity direction loss and logit-normal sampling improves U-ViT-L FID-10k from 50.22 to 37.12 and LDM-UNet from 66.73 to 60.07 (200k x 128 samples).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use FasterDiT training instead of standard DiT training | DiT-XL/2 · ImageNet 256×256, cfg=1.5, 1000k iterations | FID: 2.27 -> 2.30 | Mixed |
| Use FasterDiT training instead of standard DiT training | DiT-B/2 · ImageNet 512×512, 200k × 128 | FID-10k: 77.11 -> 58.33 | Helped |
| Use FasterDiT training instead of standard DiT training | DiT-L/2 · ImageNet 512×512, 200k × 64 | FID-10k: 67.29 -> 49.36 | Helped |
| Apply FasterDiT training method | U-ViT-L · ImageNet 512×512, 200k × 128 | FID-10k: 50.22 -> 37.12 | Helped |
| Apply FasterDiT training method | LDM-UNet · ImageNet 512×512, 200k × 128 | FID-10k: 66.73 -> 60.07 | Helped |
| Increase input standard deviation from 0.3 to 2.5 (ablation) | DDPM-linear · ImageNet 128, 100,000 iterations | FID-10k: 65.38 -> 47.04 | Helped |
| Apply lognorm(0, 1) timestep sampling instead of uniform sampling (ablation) | DDPM-linear · ImageNet 128, varying data signal intensity | FID: 47.04 -> 36.9 | Mixed |
| Replace noise prediction with velocity prediction (ablation) | Linear Flow · ImageNet 128, varying data signal intensity | FID range: 17.54 -> 11.79 | Helped |
| Replace noise prediction with velocity prediction (ablation) | Flow with cosine schedule · ImageNet 128, varying data signal intensity | FID range: 34.76 -> 12.39 | Helped |
| Add velocity direction loss to velocity MSE supervision (ablation) | FasterDiT · ImageNet 256×256 | FID | Helped |
| Add multiple-step balance (ablation) | FasterDiT · ImageNet 256×256 | FID | Helped |
| Concentrate timestep sampling with lognorm (ablation) | Linear Flow with velocity prediction · ImageNet 128, varying data signal intensity | FID | Mixed |
Surprising findings
- The best timestep-sampling/loss-weighting strategy (SD3's lognorm(-0.5,1)) is not robust: its FID becomes the worst as data signal strength is reduced, so the conclusion is data-specific.
- Flow matching with v-prediction is more robust across data std than noise prediction, with a smaller FID range (e.g., 11.79 vs 17.54 for linear flow).
Limitations
- Experiments are limited to class-conditional ImageNet; no large-scale, text-to-image, or video evaluation.
- Robustness trade-off: sharpening the timestep focus raises the performance ceiling but widens FID variance across data.
- Unclear whether findings transfer to text-conditioned architectures with joint text-visual attention (e.g., SD3).
Experimental setup
Datasets: ImageNet
Benchmarks: ImageNet 256x256 ImageNet 512x512 ImageNet 128x128
Metrics: FID FID-10k sFID IS Precision Recall
Baselines: DiT SiT MDT U-ViT LDM StyleGAN-XL ADM
Builds on: DiT SiT Stable Diffusion 3 logit-normal timestep sampling flow matching
Closest papers by meaning
- Layer- and Timestep-Adaptive Differentiable Token Compression Ratios for Efficient Diffusion Transformers (CVPR 2025 · similarity 0.90)
- TinyFusion: Diffusion Transformers Learned Shallow (CVPR 2025 · similarity 0.90)
- SparseDiT: Token Sparsification for Efficient Diffusion Transformer (NeurIPS 2025 · similarity 0.90)
- SD-DiT: Unleashing the Power of Self-supervised Discrimination in Diffusion Transformer (CVPR 2024 · similarity 0.89)
- BlockDance: Reuse Structurally Similar Spatio-Temporal Features to Accelerate Diffusion Transformers (CVPR 2025 · similarity 0.89)
- Accelerating Diffusion Transformer via Increment-Calibrated Caching with Channel-Aware Singular Value Decomposition (CVPR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.