Research map / Image generation / Fast diffusion sampling
Accelerating Diffusion Transformer via Increment-Calibrated Caching with Channel-Aware Singular Value Decomposition
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes increment-calibrated caching, a training-free DiT acceleration method that corrects cached layer outputs with low-rank (SVD-based) calibration terms, plus channel-aware SVD variants (CA-SVD, CD-SVD) to handle outlier channels.
Problem
Cache-based acceleration of diffusion transformers reuses cached features across timesteps without correction, causing quality degradation, while direct SVD compression of DiT severely hurts generation quality.
Approach
Treat low-rank SVD factors of the original linear weights as calibration parameters that add an increment term (W_a W_b (x_m - C(x))) to cached outputs, and scale weights by channel sensitivity (activation-aware or inter-timestep-delta-aware) before SVD so outlier channels are preserved.
Key results
- Versus 35-step DDIM on DiT-XL/2, ICC with 30 steps (CD-SVD, r=192) gives about 12 higher IS at under 0.06 FID increase while cutting computation by about 45%.
- Under a ~2.37T MAC budget on DiT-XL/2, ICC (CD-SVD, r=192, p=2) reaches IS 217.6 and FID 5.05 vs. naive caching IS 196.9 and FID 6.27 (p=3 setting).
- On PixArt-α with DPM 18 steps, ICC (CA-SVD, r=64) achieves FID-30k 13.94 vs. 25.47 for naive caching at a similar budget.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add increment calibration with CD-SVD instead of naive caching at a similar MAC budget | DiT-XL/2 · ImageNet 256×256 | IS: 193.33 -> 217.6 | Helped |
| Add increment calibration with CD-SVD instead of naive caching at a similar MAC budget | DiT-XL/2 · ImageNet 256×256 | FID: 6.73 -> 5.05 | Helped |
| Add channel-activation-aware SVD instead of original SVD | ICC on PixArt-α · MSCOCO-2014 | FID-30k: 9.80 -> 9.29 | Helped |
| Add channel-activation-aware SVD instead of original SVD | ICC on PixArt-α · MSCOCO-2014 | FID-30k: 18.69 -> 13.94 | Helped |
| Scale both input and output channels rather than input channels only (ablation) | CD-SVD calibration in ICC · ImageNet 256×256 | IS: 216.9 -> 217.3 | Helped |
| Scale both input and output channels rather than output channels only (ablation) | CD-SVD calibration in ICC · ImageNet 256×256 | IS: 209.1 -> 217.3 | Helped |
| Replace original SVD with CD-SVD at r=128 (ablation) | ICC on DiT-XL/2 · ImageNet 256×256 | IS: 207.1 -> 217.3 | Helped |
| Replace original SVD with CA-SVD at r=128 (ablation) | ICC on DiT-XL/2 · ImageNet 256×256 | FID: 5.11 -> 2.77 | Helped |
| Increase CA-SVD rank from 192 to 256 (ablation) | ICC on DiT-XL/2 · ImageNet 256×256 | FID: 2.74 -> 3.19 | Hurt |
| Increase CD-SVD rank from 128 to 192 (ablation) | ICC on DiT-XL/2 · ImageNet 256×256 | FID: 2.86 -> 2.53 | Helped |
Surprising findings
- Increasing the calibration rank does not monotonically improve quality; for text-to-image, FID-30k starts rising again at r=256, and some calibration settings beat non-cached sampling with the same step count.
- Direct SVD compression of DiT-XL/2 with about 33% rank removal yields large quality degradation without proportional computation savings.
Limitations
- Caching pattern is coarse-grained (FORA-style fixed period) and not optimized for finer patterns
- No single calibration method (SVD, CA-SVD, CD-SVD) consistently wins across all settings
- Quality is not monotonic in rank, requiring rank tuning
Experimental setup
Datasets: ImageNet MSCOCO-2014
Benchmarks: ImageNet 256x256 class-conditional generation MSCOCO-2014 text-to-image generation
Metrics: IS FID sFID Precision Recall FID-30k CLIP Score MACs
Baselines: FORA naive caching L2C DDIM DPM SVD
Builds on: FORA ASVD DDIM DiT
Closest papers by meaning
- Learning-to-Cache: Accelerating Diffusion Transformer via Layer Caching (NeurIPS 2024 · similarity 0.91)
- HarmoniCa: Harmonizing Training and Inference for Better Feature Caching in Diffusion Transformer Acceleration (ICML 2025 · similarity 0.91)
- Accelerating Diffusion Transformers with Token-wise Feature Caching (ICLR 2025 · similarity 0.89)
- FasterDiT: Towards Faster Diffusion Transformers Training without Architecture Modification (NeurIPS 2024 · similarity 0.89)
- CacheQuant: Comprehensively Accelerated Diffusion Models (CVPR 2025 · similarity 0.89)
- Dynamic Diffusion Transformer (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.