Research map / Video / Video generation
Timestep Embedding Tells: It's Time to Cache for Video Diffusion Model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TeaCache, a training-free caching method for DiT video diffusion models that decides when to reuse model outputs by tracking timestep-embedding-modulated noisy input differences rescaled via polynomial fitting.
Problem
Video diffusion transformers are slow at inference due to sequential denoising, and uniform output caching across timesteps ignores that output differences vary over timesteps, giving a poor efficiency-quality balance.
Approach
Use the relative L1 difference of timestep-embedding-modulated noisy inputs (available cheaply before the transformer blocks) as a proxy for output differences, rescale it with a polynomial fit to the true output differences, and cache outputs adaptively when the accumulated estimate stays below a threshold.
Key results
- On Open-Sora-Plan, TeaCache-slow reaches 4.41x speedup with VBench 80.32% vs 80.39% for the original model (about -0.07%).
- On Latte, TeaCache-fast achieves 3.28x speedup with VBench 76.69% vs 77.40% original, versus PAB-fast 1.34x.
- On Open-Sora 1.2, TeaCache-fast reaches 2.25x speedup at VBench 78.48%, while TeaCache-slow gives 1.55x with VBench 79.28%, above the original 79.22%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply TeaCache-slow caching | Latte · Latte (16 frames, 512×512) | VBench: 77.40% -> 77.40% | No clear effect |
| apply TeaCache-slow caching | Open-Sora 1.2 · Open-Sora 1.2 (51 frames, 480P) | VBench: 79.22% -> 79.28% | Helped |
| apply TeaCache-slow caching | OpenSora-Plan · OpenSora-Plan (65 frames, 512×512) | speedup: 1× -> 4.41× | Helped |
| use timestep-embedding-modulated noisy input instead of timestep embedding as the caching indicator (ablation) | TeaCache · OpenSora | VBench: 77.01% -> 78.21% | Helped |
| use timestep-embedding-modulated noisy input instead of timestep embedding as the caching indicator (ablation) | TeaCache · Latte | VBench: 77.05% -> 77.17% | Helped |
| add 1-order polynomial rescaling (ablation) | TeaCache on OpenSora · OpenSora | VBench: 78.21% -> 78.45% | Helped |
| increase polynomial fitting from 1-order to 2-order (ablation) | TeaCache on OpenSora · OpenSora | VBench: 78.45% -> 78.48% | Helped |
| increase polynomial fitting from 2-order to 4-order (ablation) | TeaCache on OpenSora · OpenSora | VBench: 78.48% -> 78.48% | No clear effect |
Surprising findings
- Timestep-embedding-modulated noisy input works better as a caching indicator than timestep embedding alone, though the noisy input by itself changes little across steps.
- On Open-Sora-Plan, PAB-fast's VBench drops to 71.81%, far below the original 80.39%, showing uniform caching can severely degrade quality.
Limitations
- Polynomial fitting coefficients are calibrated on a small set of 70 prompts and per-model, requiring model-specific fitting.
- Reference-based metrics (PSNR, SSIM) drop at extreme reduction rates, so reference-free VBench is the main quality signal.
Experimental setup
Datasets: T2V-CompBench
Benchmarks: VBench
Metrics: VBench score LPIPS SSIM PSNR latency FLOPs speedup
Baselines: PAB T-GATE Delta-DiT uniform timestep reduction
Builds on: PAB DeepCache T-GATE Delta-DiT
Closest papers by meaning
- FasterCache: Training-Free Video Diffusion Model Acceleration with High Quality (ICLR 2025 · similarity 0.90)
- MagCache: Fast Video Generation with Magnitude-Aware Cache (NeurIPS 2025 · similarity 0.90)
- VideoMLA: Low-Rank Latent KV Cache for Minute-Scale Autoregressive Video Diffusion (NeurIPS 2026 · similarity 0.89)
- DeepCache: Accelerating Diffusion Models for Free (CVPR 2024 · similarity 0.88)
- Cache Me if You Can: Accelerating Diffusion Models through Block Caching (CVPR 2024 · similarity 0.88)
- CacheQuant: Comprehensively Accelerated Diffusion Models (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.