Research map / Image generation / Fast diffusion sampling
Cache Me if You Can: Accelerating Diffusion Models through Block Caching
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Block Caching, which reuses cached outputs of spatial-transformer blocks across diffusion denoising steps (with a learned scale-shift adjustment) to speed up U-Net inference by 1.5-1.8x without quality loss.
Problem
Diffusion model image generation is costly because a large denoising U-Net must be applied many times; prior work mostly treats the network as a black box and focuses on solvers or distillation.
Approach
Analyzing per-block outputs over timesteps reveals smooth, block-specific and often tiny step-to-step changes, so blocks can reuse cached outputs from earlier steps according to an automatic threshold-based schedule, with lightweight per-timestep scale-shift parameters trained to correct cache misalignment.
Key results
- On LDM-512 with DPM, 20 steps with caching and scale-shift reaches FID 15.95 at 1.65x speedup vs 18.67 FID for the 14-step baseline at 1.43x.
- On EMU-768, human annotators preferred the caching configuration over the baseline in every comparison (e.g., 34.7% win vs 28.4% lose for DPM 20 steps).
- Caching with 50 steps and scale-shift achieves FID 15.15-15.18 at about 1.8x speedup, outperforming the 30-step baseline (FID ~17.2-17.4).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add block caching and scale-shift adjustment instead of using the 14-step baseline | LDM-512 with DPM · COCO subset (faces removed), 20 steps versus 14-step baseline | FID: 18.67 -> 15.95 | Helped |
| Add block caching and scale-shift adjustment instead of using the 14-step baseline | LDM-512 with DDIM · COCO subset (faces removed), 20 steps versus 14-step baseline | FID: 17.11 -> 16.02 | Helped |
| Add block caching and scale-shift adjustment instead of using the 30-step baseline | LDM-512 with DPM · COCO subset (faces removed), 50 steps versus 30-step baseline | FID: 17.21 -> 15.18 | Helped |
| Add block caching and scale-shift adjustment instead of using the 30-step baseline | LDM-512 with DDIM · COCO subset (faces removed), 50 steps versus 30-step baseline | FID: 17.42 -> 15.15 | Helped |
| Add scale-shift adjustment to block caching (ablation) | LDM-512 with DPM · COCO subset (faces removed), 20 steps | FID: 17.58 -> 15.95 | Helped |
| Add scale-shift adjustment to block caching (ablation) | LDM-512 with DDIM · COCO subset (faces removed), 20 steps | FID: 16.52 -> 16.02 | Helped |
| Add scale-shift adjustment to block caching (ablation) | LDM-512 with DPM · COCO subset (faces removed), 50 steps | FID: 17.23 -> 15.18 | Helped |
| Add scale-shift adjustment to block caching (ablation) | LDM-512 with DDIM · COCO subset (faces removed), 50 steps | FID: 16.65 -> 15.15 | Helped |
| Cache ResBlocks instead of SpatialTransformer blocks (ablation) | EMU-768 · DPM, 20 steps | image quality: 5% | Hurt |
| Increase the cache threshold beyond δ = 0.5 (ablation) | LDM-512 · DPM, 50 steps | image quality | Hurt |
| Set the cache threshold to δ = 0.5 (ablation) | LDM-512 · DPM, 50 steps | throughput: 1.5x | Helped |
| Add block caching | EMU-768 · DPM, 20 steps versus 14-step baseline | human visual-appeal win votes: 34.7% win, 28.4% lose | Helped |
Surprising findings
- Caching ResBlocks instead of spatial transformer blocks degrades quality significantly even for only about 5% speedup.
- Caching with scale-shift adjustment outperforms the uncached 20-step and 50-step baselines in FID, which the authors attribute to a momentum effect in the denoising trajectory.
Limitations
- Scale-shift optimization requires 12-48 hours of training on eight A100 GPUs per model and step configuration.
- Naive caching without scale-shift introduces local background artifacts.
- Caching ResBlocks is ineffective, so the method is restricted to spatial transformer blocks.
Experimental setup
Datasets: Shutterstock internal images (LDM-512 training) COCO subset (FID evaluation) OUI Prompts PartiPrompts
Benchmarks: COCO subset (FID, faces removed) OUI Prompts PartiPrompts
Metrics: FID images per second speedup human win/tie/lose votes
Baselines: Baseline with fewer steps (14 or 30) Naive caching without scale-shift DDIM DPM-Solver
Builds on: DDIM DPM-Solver Latent Diffusion Models Emu Classifier-free guidance
Closest papers by meaning
- DeepCache: Accelerating Diffusion Models for Free (CVPR 2024 · similarity 0.93)
- CacheQuant: Comprehensively Accelerated Diffusion Models (CVPR 2025 · similarity 0.93)
- Learning-to-Cache: Accelerating Diffusion Transformer via Layer Caching (NeurIPS 2024 · similarity 0.92)
- Simplified and Generalized Masked Diffusion for Discrete Data (NeurIPS 2024 · similarity 0.89)
- Faster Diffusion: Rethinking the Role of the Encoder for Diffusion Model Inference (NeurIPS 2024 · similarity 0.89)
- A Simple Early Exiting Framework for Accelerated Sampling in Diffusion Models (ICML 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.