Research map / Video / Video generation
FloVD: Optical Flow Meets Video Diffusion Model for Enhanced Camera-Controlled Video Synthesis
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
FloVD uses optical flow as the control signal for camera-controllable video diffusion, splitting flow generation into 3D-warped camera flow plus a learned object-motion model, then synthesizing video with a flow-conditioned adapter on Stable Video Diffusion.
Problem
Camera-controllable video generation typically requires ground-truth camera parameters for training, which restricts training to static-scene datasets like RealEstate10K and yields unnatural object motion and weaker generalization.
Approach
Represent camera and object motion as optical flow maps, which can be estimated from arbitrary videos without camera parameters. Camera flow is obtained by 3D warping of the input image using estimated depth; object flow is generated by a diffusion model (OMSM); the two are integrated via a segmentation mask and fed to a flow-conditioned video diffusion model (FVSM) with a T2I-Adapter-style flow e
Key results
- On RealEstate10K, FloVD trained on RE10K achieves mRotErr 1.43 and mCamMC 0.0887, versus CameraCtrl 1.44 / 0.0945 and MotionCtrl 5.90 / 0.1719.
- On Pexels-random, FloVD reports FVD 91.55 vs 151.06 for CameraCtrl and 93.54 for MotionCtrl, with FID 35.66 and IS 11.63.
- On Pexels-large (object motion >=40 px), FloVD reaches FVD 207.39 and FID 21.12 versus CameraCtrl FVD 340.36 and FID 25.53.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use FloVD instead of CameraCtrl | Camera-controlled video synthesis · RealEstate10K test | mTransErr: 0.0927 -> 0.0869 | Helped |
| Add OMSM to the baseline (ablation) | FloVD with FVSM trained on RealEstate10K · Pexels-random | FVD: 157.99 -> 104.92 | Helped |
| Add OMSM to the baseline (ablation) | FloVD with FVSM trained on RealEstate10K · Pexels-random | FID: 39.61 -> 36.33 | Helped |
| Add OMSM to the baseline (ablation) | FloVD with FVSM trained on RealEstate10K · Pexels-random | IS: 11.19 -> 11.51 | Helped |
| Train OMSM and FVSM on internal large-scale data (ablation) | FloVD + OMSM · Pexels-random | FVD: 104.92 -> 91.55 | Helped |
| Train OMSM and FVSM on internal large-scale data (ablation) | FloVD + OMSM · Pexels-random | FID: 36.33 -> 35.66 | Helped |
| Train OMSM and FVSM on internal large-scale data (ablation) | FloVD + OMSM · Pexels-random | IS: 11.51 -> 11.63 | Helped |
Limitations
- Object motion synthesis and segmentation model errors can yield unnatural object motion.
- Camera and object flow integration is not physically accurate because z-axis object motion is not captured.
- Training relies on an internal, non-released 500K-clip dataset.
Experimental setup
Datasets: internal 500K video clip dataset (Pexels-like) RealEstate10K Pexels
Benchmarks: RealEstate10K test set Pexels-random Pexels-small Pexels-medium Pexels-large
Metrics: mRotErr mTransErr mCamMC FVD FID IS
Baselines: CameraCtrl MotionCtrl
Builds on: Stable Video Diffusion T2I-Adapter Marigold CameraCtrl EDM
Closest papers by meaning
- FlowVid: Taming Imperfect Optical Flows for Consistent Video-to-Video Synthesis (CVPR 2024 · similarity 0.89)
- Wan-Move: Motion-controllable Video Generation via Latent Trajectory Guidance (NeurIPS 2025 · similarity 0.89)
- VD3D: Taming Large Video Diffusion Transformers for 3D Camera Control (ICLR 2025 · similarity 0.89)
- AC3D: Analyzing and Improving 3D Camera Control in Video Diffusion Transformers (CVPR 2025 · similarity 0.88)
- Frame In-N-Out: Unbounded Controllable Image-to-Video Generation (NeurIPS 2025 · similarity 0.88)
- Training-free Camera Control for Video Generation (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.