Research map / Image generation / Fast diffusion sampling
HarmoniCa: Harmonizing Training and Inference for Better Feature Caching in Diffusion Transformer Acceleration
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes HarmoniCa, a learning-based feature caching framework for Diffusion Transformers that trains the cache router with step-wise denoising (SDT) and an image-error-proxy objective (IEPO), achieving ~1.4-1.6x real speedups without image data.
Problem
Learning-based feature caching for Diffusion Transformers suffers from train-inference discrepancies: it ignores the effect of earlier timesteps' caching and optimizes per-step noise error rather than final image quality, which hurts quality and acceleration.
Approach
Train the caching router by simulating the full multi-step denoising trajectory with a student (cached) and teacher (uncached) model (SDT), and weight per-timestep losses by an image-error proxy computed from gradient-free generation passes (IEPO), removing the need for training images.
Key results
- On DiT-XL/2 256x256 with 10 DDIM steps, HarmoniCa reaches FID 13.35 and IS 151.83, beating Learning-to-Cache by 1.24 FID and 6.74 IS at 1.23x speedup.
- On PixArt-alpha 256x256 with 20-step DPM-Solver++, HarmoniCa achieves FID 26.33 at 1.60x speedup, below the non-accelerated FID of 27.68.
- HarmoniCa reduces training time by about 25% versus Learning-to-Cache (1.63h vs 2.15h) while using no training images.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use HarmoniCa instead of Learning-to-Cache | DiT-XL/2 · ImageNet 256×256, DDIM T=10 | FID: 14.59 -> 13.35 | Helped |
| Use HarmoniCa instead of DPM-Solver++ without caching | PixArt-α · MS-COCO 256×256, DPM-Solver++ T=20 | FID: 27.68 -> 26.33 | Helped |
| Add SDT to Learning-to-Cache training while retaining its objective (ablation) | DiT-XL/2 feature caching · ImageNet 256×256, 20-step DDIM | FID: 18.57 -> 8.01 | Helped |
| Replace the LTC objective with IEPO in SDT (ablation) | DiT-XL/2 feature caching · ImageNet 256×256, 20-step DDIM | FID: 8.01 -> 4.88 | Helped |
| Add IEPO to Learning-to-Cache training (ablation) | DiT-XL/2 feature caching · ImageNet 256×256, 20-step DDIM | FID: 18.57 -> 5.20 | Helped |
| Use the squared Frobenius image-error proxy instead of λ=1 (ablation) | SDT + IEPO · DiT-XL/2 256×256, ImageNet, 20-step DDIM | IS: 166.65 -> 206.57 | Helped |
| Change the caching threshold from 0.1 to 0.5 or 0.9 (ablation) | HarmoniCa · DiT-XL/2 256×256, ImageNet, 10-step DDIM | FID: 13.35; 13.41; 13.37 | No clear effect |
| Replace SDT with teacher forcing (ablation) | IEPO-trained DiT-XL/2 feature caching · ImageNet 256×256, 20-step DDIM | FID: 4.88 vs. 4.87 | No clear effect |
Surprising findings
- Learning-to-Cache with the even-timestep sampling implementation shows a large gap versus unconstrained sampling, e.g., IS 115.00 vs 201.37 at similar speed on DiT-XL/2.
- Sampler mismatch between training and inference (e.g., router trained with DPM-Solver++ used with SA-Solver) degrades performance relative to matched training.
Limitations
- Focuses only on block-wise caching granularity
- Evaluated only on image generation, not video/audio/3D
- Block-wise threshold beta requires per-model hyperparameter tuning (Table of beta values)
Experimental setup
Datasets: ImageNet MS-COCO
Benchmarks: ImageNet MS-COCO MJHQ-30K sDCI
Metrics: FID sFID IS Precision Recall CLIP score CUR latency DINO HPSv2
Baselines: Learning-to-Cache FORA Δ-DiT DeepCache Faster Diffusion EfficientDM PTQ4DiT Diff-Pruning DDIM DPM-Solver++ IDDPM SA-Solver
Builds on: Learning-to-Cache DDPM DDIM DPM-Solver++
Closest papers by meaning
- ProCache: Constraint-Aware Feature Caching with Selective Computation for Diffusion Transformer Acceleration (AAAI 2026 · similarity 0.94)
- Learning-to-Cache: Accelerating Diffusion Transformer via Layer Caching (NeurIPS 2024 · similarity 0.93)
- Accelerating Diffusion Transformers with Token-wise Feature Caching (ICLR 2025 · similarity 0.92)
- Accelerating Diffusion Transformer via Increment-Calibrated Caching with Channel-Aware Singular Value Decomposition (CVPR 2025 · similarity 0.91)
- BlockDance: Reuse Structurally Similar Spatio-Temporal Features to Accelerate Diffusion Transformers (CVPR 2025 · similarity 0.90)
- SparseDiT: Token Sparsification for Efficient Diffusion Transformer (NeurIPS 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.