Research map / Video / Video generation
Go-with-the-Flow: Motion-Controllable Video Diffusion Models Using Real-Time Warped Noise
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Go-with-the-Flow, a real-time noise warping algorithm that replaces temporally i.i.d. noise with flow-warped correlated noise to fine-tune video diffusion models (CogVideoX) for motion control without architecture changes.
Problem
Motion control in video diffusion models usually requires architectural changes or parameterized camera/trajectory signals, and prior noise warping (HIWYN) is too slow for large-scale video diffusion training.
Approach
Warp the initial latent noise frame-by-frame along optical flow (tracking expansion and contraction with flow density) in linear time while preserving spatial Gaussianity, then fine-tune an unmodified video diffusion model on this warped noise so motion is controlled purely through noise initialization.
Key results
- Noise warping runs at 2.14 ms at 1024x1024 on A100, about 26x faster than HIWYN (55.2 ms) while preserving spatial Gaussianity (Moran's I 0.00014, p=0.84).
- On VIPSeg local object motion control, the method achieves mIoU 0.75 and FID 41.1 versus SG-I2V 0.63 and MotionClone 0.71 (FID 61.4 and 85.2).
- User study with 40 participants gives 82% win rate for cut-and-drag object motion and 90% for turnable camera movement against baselines.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add warped-noise motion conditioning (ablation) | CogVideoX-5B · DAVIS motion transfer I2V | FID: 76.62 -> 78.6 | Helped |
| add warped-noise motion conditioning (ablation) | CogVideoX-5B · DAVIS motion transfer I2V | CoTracker mIoU: 0.52 -> 0.74 | Helped |
| increase noise degradation from γ = 0.5 to γ = 0.9 (ablation) | CogVideoX-5B motion transfer I2V · DAVIS | CoTracker mIoU: 0.74 -> 0.50 | Hurt |
| set noise degradation γ to 0.8 instead of γ = 0.5 (ablation) | CogVideoX-5B motion transfer I2V · DAVIS | FID: 78.6 -> 80.6 | Hurt |
| set noise degradation γ to 0.4 instead of γ = 0.5 (ablation) | CogVideoX-5B motion transfer I2V · DAVIS | FID: 78.6 -> 77.7 | Helped |
| set noise degradation γ to 0.2 instead of γ = 0.5 (ablation) | CogVideoX-5B motion transfer I2V · DAVIS | FID: 78.6 -> 77.1 | Helped |
| train using 33% of the dataset instead of the full dataset (ablation) | CogVideoX-5B motion transfer I2V · DAVIS | FID: 78.6 -> 100.1 | Hurt |
| train using 12.5% of the dataset instead of the full dataset (ablation) | CogVideoX-5B motion transfer I2V · DAVIS | FID: 78.6 -> 105.2 | Hurt |
| replace CogVideoX-5B with CogVideoX-2B (ablation) | Go-with-the-Flow T2V · DAVIS motion transfer | CoTracker mIoU: 0.70 -> 0.64 | Hurt |
| use Go-with-the-Flow instead of DMT | DAVIS motion transfer T2V · DAVIS | CoTracker mIoU: 0.85 -> 0.70 | Hurt |
| use Go-with-the-Flow instead of MotionClone | motion transfer I2V · DL3DV camera movement transfer | FID: 82.7 -> 48.4 | Helped |
| use Go-with-the-Flow instead of MotionClone | motion transfer I2V · DL3DV camera movement transfer | CoTracker mIoU: 0.71 -> 0.83 | Helped |
Surprising findings
- Removing temporal Gaussianity from the noise distribution does not deteriorate video diffusion fine-tuning; the model adapts quickly to warped noise.
- Warped noise with bilinear/bicubic/nearest interpolation yields low warping error but severely degraded image quality due to spatial non-Gaussianity.
Limitations
- Relies on optical flow quality (RAFT) and user polygon approximations for object control.
- Motion strength requires tuning the degradation level gamma per task.
Experimental setup
Datasets: DAVIS DL3DV VIPSeg WonderJourney videos
Benchmarks: DAVIS VIPSeg DL3DV WonderJourney VBench DeepFloyd IF super-resolution DifFRelight relighting
Metrics: Moran's I K-S test GPU time LPIPS SSIM PSNR warping error FID FVD CoTracker mIoU
Baselines: MotionClone SG-I2V DragAnything DMT MotionCtrl ImageConductor AnyV2V HIWYN InfRes PYoCo Control-A-Video
Builds on: HIWYN CogVideoX RAFT AnimateDiff InfRes
Closest papers by meaning
- FlowMo: Variance-Based Flow Guidance for Coherent Motion in Video Generation (NeurIPS 2025 · similarity 0.89)
- TRIP: Temporal Residual Learning with Image Noise Prior for Image-to-Video Diffusion Models (CVPR 2024 · similarity 0.88)
- Training-free Camera Control for Video Generation (ICLR 2025 · similarity 0.88)
- Motion-Zero: A Zero-Shot Trajectory Control Framework of Moving Object for Diffusion-Based Video Generation (AAAI 2025 · similarity 0.88)
- Wan-Move: Motion-controllable Video Generation via Latent Trajectory Guidance (NeurIPS 2025 · similarity 0.88)
- Identifying and Solving Conditional Image Leakage in Image-to-Video Diffusion Model (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.