Research map / Video / Video generation
Foresight: Adaptive Layer Reuse for Accelerated and High-Quality Text-to-Video Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Foresight is a training-free, per-layer adaptive block-reuse scheduler that decides at runtime whether to recompute or reuse DiT block outputs in text-to-video diffusion, using MSE-based thresholds from a warmup phase.
Problem
Video diffusion transformers are expensive to run because of quadratic spatial-temporal attention repeated over many denoising steps, and existing static caching reuses features with a uniform policy that degrades quality.
Approach
Instead of a fixed reuse schedule, Foresight sets per-layer reuse thresholds from MSE of block outputs during a warmup phase and dynamically reuses or recomputes each spatial/temporal DiT block at each step, caching only full DiT block outputs (coarse-grained) to also lower memory.
Key results
- On Open-Sora, Foresight (N1R2) reaches VBench 75.90 and PSNR 29.67 versus PAB's 75.32 and 25.67, with a 1.28x speedup.
- On CogVideoX, Foresight (N2R3) achieves a 1.63x speedup with VBench 77.84, while N1R2 gives PSNR 34.75 and FVD 130.65 versus PAB's 29.04 and 340.24.
- Foresight reports up to 1.8x-2.2x speedups on HunyuanVideo and Wan-2.1 in the appendix, with a 3x memory reduction from block-level caching.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use adaptive layer reuse (N1R2) instead of PAB | Open-Sora · VBench prompt set | VBench (%): 75.32 -> 75.90 | Helped |
| Use adaptive layer reuse (N1R2) instead of PAB | Latte · VBench prompt set | VBench (%): 72.64 -> 73.08 | Helped |
| Use adaptive layer reuse (N1R2) instead of PAB | CogVideoX · VBench prompt set | VBench (%): 77.89 -> 77.94 | Helped |
| Increase reuse settings from N=1, R=2 to N=2, R=3 (ablation) | Open-Sora · VBench prompt set | Speedup: 1.28× -> 1.44× | Helped |
| Increase reuse settings from N=1, R=2 to N=2, R=3 (ablation) | Latte · VBench prompt set | Speedup: 1.18× -> 1.31× | Helped |
| Increase reuse settings from N=1, R=2 to N=2, R=3 (ablation) | CogVideoX · VBench prompt set | Speedup: 1.46× -> 1.63× | Helped |
| Increase reuse settings from N=1, R=2 to N=2, R=3 (ablation) | Open-Sora · VBench prompt set | VBench (%): 75.90 -> 75.62 | No clear effect |
| Increase reuse window and compute interval from N=1,R=2 to N=4,R=5 (ablation) | Foresight on Open-Sora · Open-Sora (240p, 2s, T=60) | Latency: 18.70 -> 13.49 | Helped |
| Increase reuse window and compute interval from N=1,R=2 to N=4,R=5 (ablation) | Foresight on Open-Sora · Open-Sora (240p, 2s, T=60) | PSNR: 32.38 -> 27.91 | Hurt |
| Set gamma=0.25 instead of gamma=0.5 (ablation) | Foresight on Open-Sora · Open-Sora (240p, 2s, T=60) | PSNR: 32.38 -> 38.09 | Helped |
| Set gamma=0.25 instead of gamma=0.5 (ablation) | Foresight on Open-Sora · Open-Sora (240p, 2s, T=60) | Latency: 18.70 -> 20.50 | Hurt |
Surprising findings
- Static reuse on CogVideoX drops VBench from 77.78 to 54.16 for Delta-DiT, showing that existing static caches can severely degrade quality.
- Larger reuse windows (N>=3) eventually lower quality relative to PAB because the reuse metric is not updated often enough.
Limitations
- Speed and quality depend on the reuse window N and warmup length W, and larger N or smaller W trade quality for speed.
- Experiments use only coarse block-level caching; finer granularity is left unexplored.
- Several speedup numbers vary notably across configurations and latency standard deviations are large relative to some gains.
Experimental setup
Datasets: VBench prompt set UCF-101 prompts EvalCrafter prompts Open-Sora prompt set
Benchmarks: VBench UCF-101 EvalCrafter PenguinVideoBenchmark
Metrics: VBench accuracy PSNR SSIM LPIPS FVD latency speedup CLIPSIM CLIP-Temp DOVER VQA
Baselines: Static caching Delta-DiT T-GATE PAB TeaCache
Builds on: PAB DeepCache T-GATE Delta-DiT TeaCache
Closest papers by meaning
- FIFO-Diffusion: Generating Infinite Videos from Text without Training (NeurIPS 2024 · similarity 0.86)
- FlexiDiT: Your Diffusion Transformer Can Easily Generate High-Quality Samples with Less Compute (CVPR 2025 · similarity 0.86)
- Training-Free Efficient Video Generation via Dynamic Token Carving (NeurIPS 2025 · similarity 0.86)
- Towards a Golden Classifier-Free Guidance Path via Foresight Fixed Point Iterations (NeurIPS 2025 · similarity 0.86)
- FreeNoise: Tuning-Free Longer Video Diffusion via Noise Rescheduling (ICLR 2024 · similarity 0.85)
- AdaDiff: Adaptive Step Selection for Fast Diffusion Models (AAAI 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.