Research map / Architectures and efficiency / Attention and transformers
Learning-to-Cache: Accelerating Diffusion Transformer via Layer Caching
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Learning-to-Cache (L2C) learns a continuous, input-invariant, timestep-variant router that decides which diffusion transformer layers to reuse from the previous step, training only the router while keeping model weights frozen.
Problem
Diffusion transformers have slow inference because every denoising step runs a large transformer; naive layer dropping degrades quality and exhaustive search over layer caching choices is exponential in depth.
Approach
Interpolate each transformer layer's residual branch between the outputs at two adjacent timesteps, make the binary cache/compute choice per layer and timestep differentiable via a learnable router beta, and optimize beta with a regularized approximation loss while freezing the diffusion model; the discretized router yields a static computation graph for inference.
Key results
- With 20-step DDIM on DiT-XL/2, L2C achieves FID 3.46 at 1.27x speedup versus 4.68 FID for 16-step DDIM at 1.25x speedup.
- On U-ViT-H/2, up to 93.68% of layers can be removed in cache steps (46.84% overall at 50 NFE) with FID change below 0.01.
- Compared with DeepCache on U-ViT-H/2 at 20 NFE, L2C reaches FID 2.64 vs 2.70 at similar speedup (1.67x vs 1.64x).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use layer caching instead of fewer-step DDIM at 20 NFE-comparable speed | DiT-XL/2 · ImageNet 256×256, 20-step comparison | FID: 4.68 -> 3.46 | Helped |
| Use layer caching instead of fewer-step DDIM | DiT-XL/2 · ImageNet 256×256, 10 NFE | FID: 16.57 -> 12.79 | Helped |
| Use layer caching instead of fewer-step DDIM | DiT-XL/2 · ImageNet 512×512, 50 NFE | FID: 3.28 -> 3.69 | Hurt |
| Use layer caching instead of DPM-Solver | U-ViT-H/2 · ImageNet 256×256, 50 NFE | FID: 2.3728 -> 2.3625 | Helped |
| Use layer caching instead of DPM-Solver | U-ViT-H/2 · ImageNet 256×256, 20 NFE | FID: 2.5739 -> 2.5809 | Hurt |
| Shift cache steps from [2,4,6,8,10,...] to [3,5,7,9,11,...] (ablation) | DPM-Solver-2 with all layers cached · ImageNet 256×256, 20 NFE | FID: 5.30 -> 2.80 | Helped |
| Cache layers rather than learn to drop layers (ablation) | DiT-XL/2 · ImageNet 256×256, 10 NFE | FID: 17.35 -> 3.47 | Helped |
| Cache layers rather than randomly drop layers (ablation) | DiT-XL/2 · ImageNet 256×256, 10 NFE | FID: 277.42 -> 3.47 | Helped |
| Increase λ from 0 to 5e-6 (ablation) | L2C router · ImageNet 256×256, 10 NFE | FID: 3.48 -> 3.45 | Helped |
| Increase λ from 5e-6 to 1e-4 (ablation) | L2C router · ImageNet 256×256, 10 NFE | FID: 3.45 -> 6.51 | Hurt |
| Use a larger threshold at higher acceleration ratios (ablation) | L2C router · Image generation; higher acceleration ratios | image quality | Helped |
| Use a smaller threshold at lower acceleration ratios (ablation) | L2C router · Image generation; lower acceleration ratios | image quality | Helped |
Surprising findings
- Dropping layers without caching severely degrades DiT images (random drop FID 277 at 1.18x), while caching the same number of layers preserves quality.
- Learned caching patterns differ strongly across architectures: the middle of U-ViT is almost fully cacheable, while no clear pattern emerges for DiT-XL.
Limitations
- Acceleration is capped at 2x because caching is applied every other step.
- Performance depends on the trained model; a slight FID drop occurs for DiT-XL/2 at 512 resolution and lossless caching is not uniform across models.
- Lower acceleration ratios and fewer-step settings show less lossless caching potential.
Experimental setup
Datasets: ImageNet
Benchmarks: ImageNet 256x256 ImageNet 512x512
Metrics: FID sFID IS Precision Recall MACs latency speedup
Baselines: DDIM DPM-Solver DeepCache Faster Diffusion Random Drop Learning-to-Drop
Builds on: DDIM DPM-Solver DeepCache Faster Diffusion
Closest papers by meaning
- HarmoniCa: Harmonizing Training and Inference for Better Feature Caching in Diffusion Transformer Acceleration (ICML 2025 · similarity 0.93)
- Accelerating Diffusion Transformers with Token-wise Feature Caching (ICLR 2025 · similarity 0.92)
- Cache Me if You Can: Accelerating Diffusion Models through Block Caching (CVPR 2024 · similarity 0.92)
- Accelerating Diffusion Transformer via Increment-Calibrated Caching with Channel-Aware Singular Value Decomposition (CVPR 2025 · similarity 0.91)
- CacheQuant: Comprehensively Accelerated Diffusion Models (CVPR 2025 · similarity 0.91)
- DeepCache: Accelerating Diffusion Models for Free (CVPR 2024 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.