Research map / Video / Video generation
Fast and Memory-Efficient Video Diffusion Using Streamlined Inference
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Streamlined Inference, a training-free framework (Feature Slicer, Operator Grouping with pipelining, and Step Rehash) that cuts peak memory and latency of video diffusion models such as SVD, SVD-XT, and AnimateDiff.
Problem
Video diffusion models have high computation and peak GPU memory costs, especially for longer and higher-resolution videos, which prevents running them on consumer GPUs; existing compression methods need retraining and naive clip-by-clip slicing causes artifacts.
Approach
Losslessly slice spatial and temporal features into sub-features, group consecutive homogeneous operators so each sub-feature flows through them without full intermediate maps, pipeline the groups for parallelism, and reuse temporal-layer outputs of the final up-sampling block across adjacent denoising steps (Step Rehash).
Key results
- On AnimateDiff at 576x1024, peak memory drops from 41.71G to 11.07G with latency 24.38s to 15.15s while FVD on UCF-101 moves from 758.7 to 784.5.
- On SVD at 576x1024, peak memory drops from 39.49G to 23.42G and latency from 23.29s to 14.24s, with FVD 307.7 to 340.6.
- Step Rehash outperforms DeepCache at the same 13 full-computation steps on UCF-101 (SVD FVD 340.6 vs 394.0; AnimateDiff FVD 784.5 vs 840.2).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace original inference with Streamlined Inference | SVD · UCF-101 | FVD: 307.7 -> 340.6 | Hurt |
| replace original inference with Streamlined Inference | SVD · 576 × 1024 | Latency: 23.29s -> 14.24s | Helped |
| replace original inference with Streamlined Inference | SVD-XT · UCF-101 | FVD: 387.9 -> 424.7 | Hurt |
| replace original inference with Streamlined Inference | SVD-XT · 576 × 1024 | Latency: 40.77s -> 25.47s | Helped |
| replace original inference with Streamlined Inference | AnimateDiff · UCF-101 | FVD: 758.7 -> 784.5 | Hurt |
| replace original inference with Streamlined Inference | AnimateDiff · 576 × 1024 | Latency: 24.38s -> 15.15s | Helped |
| replace original inference with Streamlined Inference | AnimateDiff · 576 × 1024 | Peak memory: 41.71G -> 11.07G | Hurt |
| replace original inference with Streamlined Inference | SVD · 576 × 1024 | Peak memory: 39.49G -> 23.42G | Hurt |
| use Step Rehash instead of original inference (ablation) | SVD · UCF-101 | FVD: 307.7 -> 340.6 | Hurt |
| use Step Rehash instead of original inference (ablation) | AnimateDiff · MSR-VTT | FVD: 607.1 -> 603.9 | Helped |
| use Step Rehash instead of DeepCache (ablation) | SVD · UCF-101 | FVD: 394.0 -> 340.6 | Helped |
| use Step Rehash instead of DeepCache (ablation) | AnimateDiff · MSR-VTT | FVD: 615.8 -> 603.9 | Helped |
Surprising findings
- Naive clip-by-clip slicing causes large quality loss (AnimateDiff FVD 2403.9 vs 758.7) and increases latency, so slicing alone is not sufficient.
- Operator grouping without pipelining was about 10% slower than the original unsliced version, attributed to the memory-bound nature of inference.
Limitations
- Efficiency is bounded by the baseline model architecture.
- Pipeline implementation is not released because it needs specific compilation support.
- Slight FVD increase relative to the original on some settings (e.g., SVD 307.7 to 340.6).
Experimental setup
Datasets: UCF-101 MSR-VTT
Benchmarks: UCF-101 MSR-VTT
Metrics: FVD CLIP-Score peak memory latency
Baselines: Original inference Naive Slicing DeepCache
Builds on: DeepCache
Closest papers by meaning
- FasterCache: Training-Free Video Diffusion Model Acceleration with High Quality (ICLR 2025 · similarity 0.90)
- Towards Precise Scaling Laws for Video Diffusion Transformers (CVPR 2025 · similarity 0.89)
- From Slow Bidirectional to Fast Autoregressive Video Diffusion Models (CVPR 2025 · similarity 0.89)
- AsymRnR: Video Diffusion Transformers Acceleration with Asymmetric Reduction and Restoration (ICML 2025 · similarity 0.89)
- IV-mixed Sampler: Leveraging Image Diffusion Models for Enhanced Video Synthesis (ICLR 2025 · similarity 0.89)
- AR-Diffusion: Asynchronous Video Generation with Auto-Regressive Diffusion (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.