Research map / Video / Video generation
Collaborative Video Diffusion: Consistent Multi-video Generation with Camera Control
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Collaborative Video Diffusion (CVD), which fine-tunes CameraCtrl/AnimateDiff with an epipolar-attention cross-video synchronization module so multiple camera-controlled videos of the same scene stay consistent.
Problem
Generating multiple videos of the same scene from different camera trajectories with consistent content, geometry, and motion; existing camera-controlled video models generate each video independently.
Approach
A cross-video synchronization module applies epipolar-masked attention between corresponding frames of paired videos (using ground-truth fundamental matrices or pseudo-epipolar lines), trained with a hybrid scheme on static RealEstate10K multi-view clips and homography-augmented WebVid10M monocular clips; an inference procedure extends the pairwise model to arbitrary numbers of videos.
Key results
- On RE10K, CVD reaches 55.5/71.8/83.3 rotation AUC and 76.9 precision, beating MotionCtrl (49.0/68.0/81.2, precision 64.6) and CameraCtrl.
- On WebVid10M cross-video geometry, CVD reaches rotation AUC 25.2/40.7/57.5 versus 12.2/28.2/48.0 for MotionCtrl+SVD.
- CVD achieves CLIP-F 0.93 versus 0.79-0.86 for baselines, and FID 32.90 (comparable to CameraCtrl's 32.10).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use CVD with epipolar cross-video attention instead of the strongest reported baseline | Camera-controlled video generation · RE10K | Rot. AUC (@5°/10°/20°): 49.0 / 68.0 / 81.2 -> 55.5 / 71.8 / 83.3 | Helped |
| use CVD instead of the strongest reported baseline | Camera-controlled video generation · WV10M | Rot. AUC (@5°/10°/20°): 12.2 / 28.2 / 48.0 -> 25.2 / 40.7 / 57.5 | Helped |
| use CVD rather than AnimateDiff+SparseCtrl | Camera-controlled video generation · WV10M | CLIP-F: 0.86 -> 0.93 | Helped |
| use CVD rather than CameraCtrl | Camera-controlled video generation · RE10K | FID: 32.10 -> 32.90 | Hurt |
| replace epipolar attention with normal self-attention (ablation) | CVD · WV10M generic scenes | Rot. AUC (@5°/10°/20°): 25.2 / 40.7 / 57.5 -> 16.8 / 31.8 / 49.1 | Hurt |
| train only on RealEstate10K instead of using mixed training (ablation) | CVD · WV10M generic scenes | CLIP-F: 0.90 -> 0.93 | Hurt |
| remove homography augmentation from WebVid10M videos (ablation) | CVD · WV10M generic scenes | Rot. AUC (@5°/10°/20°): 22.0 / 35.5 / 50.5 -> 25.2 / 40.7 / 57.5 | Hurt |
| use only 1 Cross-View Synchronization Module instead of 2 (ablation) | CVD · WV10M generic scenes | Rot. AUC (@5°/10°/20°): 22.7 / 37.8 / 54.3 -> 25.2 / 40.7 / 57.5 | Hurt |
Surprising findings
- Removing epipolar constraints keeps semantic consistency high (CLIP-F 0.91) but substantially lowers geometric consistency, indicating semantic attention alone does not ensure geometry.
Limitations
- Performance is bounded by base models AnimateDiff and CameraCtrl, inheriting shape shifting and dynamic inconsistencies.
- Does not solve internal consistency within individual videos.
- Not real-time due to diffusion cost.
Experimental setup
Datasets: RealEstate10K WebVid10M
Benchmarks: RealEstate10K WebVid10M
Metrics: Rotation AUC Translation AUC SuperGlue precision SuperGlue matching score CLIP-T CLIP-F FID KID
Baselines: CameraCtrl MotionCtrl CameraCtrl+SparseCtrl MotionCtrl+SVD AnimateDiff+SparseCtrl
Builds on: CameraCtrl AnimateDiff SparseCtrl Stable Diffusion SuperGlue
Closest papers by meaning
- VD3D: Taming Large Video Diffusion Transformers for 3D Camera Control (ICLR 2025 · similarity 0.89)
- SynCamMaster: Synchronizing Multi-Camera Video Generation from Diverse Viewpoints (ICLR 2025 · similarity 0.88)
- DiTCtrl: Exploring Attention Control in Multi-Modal Diffusion Transformer for Tuning-Free Multi-Prompt Longer Video Generation (CVPR 2025 · similarity 0.88)
- Modular-Cam: Modular Dynamic Camera-view Video Generation with LLM (AAAI 2025 · similarity 0.87)
- LongDiff: Training-Free Long Video Generation in One Go (CVPR 2025 · similarity 0.87)
- Training-free Camera Control for Video Generation (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.