Research map / Image generation / Fast diffusion sampling
CacheQuant: Comprehensively Accelerated Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CacheQuant, a training-free framework that jointly optimizes diffusion model caching and quantization via dynamic-programming cache scheduling and decoupled error correction.
Problem
Diffusion models are slow and structurally complex due to redundancy at both temporal and structural levels; combining caching and quantization naively causes compounded errors and large performance degradation.
Approach
Jointly optimize cache schedule and quantization by framing cache-group selection as dynamic programming over caching plus quantization L1 errors, then apply decoupled (cache vs. quantization) channel-wise affine error correction per step, absorbed into weight quantization, without retraining.
Key results
- On Stable Diffusion with MS-COCO, CacheQuant (W8A8, N=5) achieves 5.18× speedup and 4× compression with CLIP score 26.87 vs 26.89 for the 50-step PLMS baseline.
- On ImageNet LDM-4 (W8A8, N=20), CacheQuant reaches FID 7.21 at 18.06× speedup versus 8.08 for DeepCache at 10.54× (full precision DeepCache N=20).
- Ablation on ImageNet shows 8-bit quantized DeepCache baseline FID 15.36 improves to 8.47 with DPS, 7.21 with DPS+EDC, and 6.34 with reconstruction.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| jointly apply caching and quantization (N=5, 8/8) | Stable Diffusion · MS-COCO | CLIP Score: 26.89 -> 26.87 | No clear effect |
| jointly apply caching and quantization (N=5, 8/8) | Stable Diffusion · MS-COCO | speedup: 5.18× | Helped |
| add DPS to the 8/8 Deepcache-N=20 baseline (ablation) | LDM-4 · ImageNet 256 × 256 | FID: 15.36 -> 8.47 | Helped |
| add EDC to DPS (ablation) | LDM-4 · ImageNet 256 × 256 | FID: 8.47 -> 7.21 | Helped |
| add reconstruction to DPS+EDC (ablation) | LDM-4 · ImageNet 256 × 256 | FID: 7.21 -> 6.34 | Helped |
| add reconstruction to DPS+EDC (ablation) | LDM-4 · ImageNet 256 × 256 | IS: 160.68 -> 180.42 | Helped |
| replace uniform caching schedule with DPS-selected schedule (ablation) | CacheQuant · ImageNet 256 × 256 | FID: 15.36 -> 8.47 | Helped |
| use cache frequency N=20 instead of N=5 (ablation) | LDM-4 with CacheQuant (W8A8) · ImageNet 256 × 256 | FID: 4.03 -> 7.21 | Hurt |
| use cache frequency N=20 instead of N=5 (ablation) | LDM-4 with CacheQuant (W8A8) · ImageNet 256 × 256 | speedup: 7.87× -> 18.06× | Helped |
| combine caching and quantization without joint optimization | LDM · ImageNet | FID loss: -0.76; -4.71; -11.99 | Hurt |
| use DEC instead of direct correction (ablation) | LDM · ImageNet | FID: +0.91 | Helped |
| reduce PLMS sampling steps from 50 to 20 | Stable Diffusion · MS-COCO | FID: 25.59 -> 24.70 | No clear effect |
Surprising findings
- Naively integrating caching and quantization yields FID loss of 11.99, much worse than either alone (0.76 and 4.71), showing the two optimizations are not orthogonal.
Limitations
- Speedup measured on GPU is much larger than on CPU and ARM, so real-world gains depend on hardware.
- Relies on the base temporal quantizer from prior work and cache block choices tuned per model (e.g., last blocks, middle blocks).
Experimental setup
Datasets: CIFAR-10 LSUN-Bedroom LSUN-Church ImageNet MS-COCO PartiPrompt
Benchmarks: CIFAR-10 LSUN-Church LSUN-Bedroom ImageNet MS-COCO PartiPrompt
Metrics: FID IS CLIP Score Bops speedup model size
Baselines: DeepCache Δ-DiT EDA-DM Diff-Pruning Small SD BK-SDM PLMS
Builds on: DeepCache Δ-DiT EDA-DM
Closest papers by meaning
- DeepCache: Accelerating Diffusion Models for Free (CVPR 2024 · similarity 0.94)
- Cache Me if You Can: Accelerating Diffusion Models through Block Caching (CVPR 2024 · similarity 0.93)
- A Simple Early Exiting Framework for Accelerated Sampling in Diffusion Models (ICML 2024 · similarity 0.91)
- Learning-to-Cache: Accelerating Diffusion Transformer via Layer Caching (NeurIPS 2024 · similarity 0.91)
- ALTER: All-in-One Layer Pruning and Temporal Expert Routing for Efficient Diffusion Generation (NeurIPS 2025 · similarity 0.90)
- Analyzing and Improving the Training Dynamics of Diffusion Models (CVPR 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.