Research map / Video / Video generation
FlowVid: Taming Imperfect Optical Flows for Consistent Video-to-Video Synthesis
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
FlowVid trains a flow-warped, depth-conditioned video diffusion model on 100k Shutterstock videos and performs video-to-video editing by editing the first frame with an off-the-shelf I2I model and propagating edits via flow warping.
Problem
Text-guided video-to-video synthesis suffers from temporal inconsistency (flicker); prior methods using optical flow treat estimated flow as a hard constraint, which fails when flow estimates are inaccurate.
Approach
Use imperfect optical flow as a soft condition: warp the first frame along estimated flow to form a flow-warped video, feed it with spatial controls (depth/canny) into an inflated ControlNet video diffusion model, and generate by editing only the first frame with any I2I model and propagating edits.
Key results
- User study preference: FlowVid 45.7% vs TokenFlow 40.4%, Rerender 10.2%, CoDeF 3.5%.
- Generates 120 frames (4s at 30 FPS) at 512x512 in about 1.5 minutes on one A100, 3.1x to 10.5x faster than baselines.
- Ablation: spatial-only condition wins 9% vs full model; adding flow-warped video raises it to 38%, and adding occlusion mask and first frame reaches 42%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use FlowVid instead of TokenFlow | Video-to-video synthesis · DAVIS user study | preference rate: 40.4 ± 5.3 -> 45.7 ± 6.4 | Helped |
| use FlowVid instead of Rerender | Video-to-video synthesis · DAVIS user study | preference rate: 10.2 ± 7.1 -> 45.7 ± 6.4 | Helped |
| use FlowVid instead of CoDeF | Video-to-video synthesis · DAVIS user study | preference rate: 3.5 ± 1.9 -> 45.7 ± 6.4 | Helped |
| use FlowVid instead of TokenFlow | 4-second 512×512 video synthesis · 4-second 512×512 video | runtime: 15.8 -> 1.5 | Helped |
| use FlowVid instead of Rerender | 4-second 512×512 video synthesis · 4-second 512×512 video | runtime: 10.8 -> 1.5 | Helped |
| use FlowVid instead of CoDeF | 4-second 512×512 video synthesis · 4-second 512×512 video | runtime: 4.6 -> 1.5 | Helped |
| add flow-warped video to spatial controls (ablation) | FlowVid · Condition combinations | winning rate against full FlowVid: 9% -> 38% | Helped |
| add flow occlusion masks to spatial controls and flow-warped video (ablation) | FlowVid · Condition combinations | winning rate against full FlowVid: 38% -> 42 % | Helped |
| add the first-frame condition to spatial controls, flow-warped video, and flow occlusion (ablation) | FlowVid · Condition combinations | winning rate against full FlowVid: 42 % -> full FlowVid | Helped |
| replace UniMatch with RAFT optical flow (ablation) | FlowVid · DAVIS user study | user preference: 28.7; 45.2; 26.1 | No clear effect |
| use canny-edge spatial controls instead of depth maps (ablation) | FlowVid · Stylization and object swap | editing quality | Mixed |
Surprising findings
- Using lower-quality RAFT flow instead of UniMatch gave similar user preferences, suggesting robustness to flow quality.
Limitations
- Relies heavily on the quality of the edited first frame; structural misalignment propagates to later frames.
- Large occlusions from fast motion cause hallucinated or color-shifted regions.
Experimental setup
Datasets: Shutterstock videos (100k) DAVIS (25 videos for evaluation)
Benchmarks: DAVIS (25 videos, 115 prompts)
Metrics: user preference rate runtime (minutes)
Baselines: Rerender CoDeF TokenFlow ControlNet per-frame
Builds on: ControlNet latent diffusion models Tune-A-Video FateZero UniMatch
Closest papers by meaning
- FloVD: Optical Flow Meets Video Diffusion Model for Enhanced Camera-Controlled Video Synthesis (CVPR 2025 · similarity 0.89)
- Looking Backward: Streaming Video-to-Video Translation with Feature Banks (ICLR 2025 · similarity 0.88)
- Hierarchical Flow Diffusion for Efficient Frame Interpolation (CVPR 2025 · similarity 0.88)
- FlowMo: Variance-Based Flow Guidance for Coherent Motion in Video Generation (NeurIPS 2025 · similarity 0.88)
- BIVDiff: A Training-Free Framework for General-Purpose Video Synthesis via Bridging Image and Video Diffusion Models (CVPR 2024 · similarity 0.88)
- StreamFlow: Streamlined Multi-Frame Optical Flow Estimation for Video Sequences (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.