Research map / Video / Video generation
Looking Backward: Streaming Video-to-Video Translation with Feature Banks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces StreamV2V, a training-free streaming video-to-video translation method on StreamDiffusion/LCM that uses a dynamically merged feature bank with extended self-attention and feature fusion to reach 20 FPS on one A100.
Problem
Batch-based diffusion video-to-video translation is limited to about 4-second clips, needs minutes per clip, and cannot run in real time on streaming input; per-frame image diffusion flickers.
Approach
A backward-looking feature bank of past-frame keys, values, and output features, read via extended self-attention and cosine-similarity feature fusion, and kept compact by dynamically merging stored and new tokens with bipartite matching, so memory stays O(1) in video length.
Key results
- StreamV2V runs at 20 FPS on one A100 at 512x512, reported as 15x, 46x, 108x, and 158x faster than FlowVid, CoDeF, Rerender, and TokenFlow.
- Warp error of 102.99 is the best among compared methods, versus 117.01 for StreamDiffusion; CLIP score is 96.58, third behind TokenFlow (97.04) and FlowVid (96.68).
- User study: StreamV2V wins over 70% against StreamDiffusion and over 80% against CoDeF, but loses to Rerender, FlowVid, and TokenFlow.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add the feature bank with Extended self-Attention (ablation) | StreamDiffusion · 18 in-house videos | warp error: 85.2 -> 74.0 | Helped |
| add Feature Fusion without Extended self-Attention (ablation) | StreamDiffusion · 18 in-house videos | warp error: 85.2 -> 80.4 | Helped |
| add Feature Fusion to Extended self-Attention (ablation) | StreamV2V · 18 in-house videos | warp error: 74.0 -> 73.4 | Helped |
| replace the queue bank with a size-1 dynamic merging bank (ablation) | StreamV2V · 18 in-house videos | warp error: queue (1): 76.4; queue (2): 74.8; queue (4): 72.4; DyMe (1): 73.4 | Mixed |
| increase queue bank size from 1 to 2 to 4 (ablation) | StreamV2V · 18 in-house videos, 512×512 | time (ms): 70 -> 79 -> 100 | Hurt |
| vary Feature Fusion similarity threshold from 0.9 to 0.0 or 1.0 (ablation) | StreamV2V · 19 videos | warp error: 73.4 -> 74.0 | Hurt |
| set the bank updating interval to 8 instead of the default 4 (ablation) | StreamV2V · 19 videos | warp error: 73.4 -> 72.4 | Helped |
| update the bank every 1 or 2 frames instead of every 4 (ablation) | StreamV2V · 19 videos | warp error: 73.4 -> 78.1; 73.4 -> 75.3 | Hurt |
| update the bank every 16 frames instead of every 4 (ablation) | StreamV2V · 19 videos | warp error: 73.4 -> 72.5 | Helped |
| use uniform grid sampling instead of random sampling for DyMe (ablation) | StreamV2V · 19 videos | warp error: 73.4 -> 73.7 | Hurt |
| use split sampling instead of random sampling for DyMe (ablation) | StreamV2V · 19 videos | warp error: 73.4 -> 74.2 | Hurt |
| reduce LCM denoising steps (ablation) | StreamV2V · video generation | video quality | Hurt |
Surprising findings
- The feature bank can be compressed to the size of a single frame (DyMe bank of size 1) while still outperforming a queue bank of size 2 in warp error.
- Applying feature fusion at all U-Net layers causes blurring, so restricting it to low-resolution blocks works better.
Limitations
- Fails to substantially alter the subject for some prompts (e.g., Pope, Batman) because SDEdit has limited editing ability.
- Produces inconsistent output with rapid camera or object motion, and still trails FlowVid and TokenFlow in quality.
- Memory overhead of the DyMe bank grows relative to baseline at higher resolutions (+136% at 1536x1536).
Experimental setup
Datasets: DAVIS 2017
Benchmarks: DAVIS 2017
Metrics: CLIP score warp error user preference win rate runtime (ms per frame / FPS) memory (GiB)
Baselines: StreamDiffusion CoDeF Rerender TokenFlow FlowVid
Builds on: StreamDiffusion Latent Consistency Model SDEdit Stable Diffusion bipartite matching token merging
Closest papers by meaning
- From Slow Bidirectional to Fast Autoregressive Video Diffusion Models (CVPR 2025 · similarity 0.89)
- FlowVid: Taming Imperfect Optical Flows for Consistent Video-to-Video Synthesis (CVPR 2024 · similarity 0.88)
- StreamingT2V: Consistent, Dynamic, and Extendable Long Video Generation from Text (CVPR 2025 · similarity 0.88)
- VideoMLA: Low-Rank Latent KV Cache for Minute-Scale Autoregressive Video Diffusion (NeurIPS 2026 · similarity 0.87)
- MeDM: Mediating Image Diffusion Models for Video-to-Video Translation with Temporal Correspondence Guidance (AAAI 2024 · similarity 0.87)
- Frame Context Packing and Drift Prevention in Next-Frame-Prediction Video Diffusion Models (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.