Research map / Video / Video generation
Cavia: Camera-controllable Multi-view Video Diffusion with View-Integrated Attention
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Cavia extends Stable Video Diffusion with cross-view and cross-frame 3D attention and jointly trains on static scenes, synthetic Objaverse multi-view dynamic objects, and filtered monocular videos to generate camera-controllable, multi-view consistent videos from one image.
Problem
Image-to-video diffusion models lack precise camera controllability and 3D consistency, and independently sampled videos of the same scene from different camera paths are inconsistent; multi-view dynamic video data is scarce.
Approach
Inflate SVD's spatial and temporal attention into cross-view and cross-frame 3D attention (by rearranging view/frame/spatial tokens), which preserves pretrained weights and lets the model train jointly on static multi-view scenes, synthetic animated Objaverse objects, and pose-annotated monocular videos with Plücker camera conditioning.
Key results
- On RealEstate10K monocular evaluation, Cavia reports COLMAP error 14.4% vs 19.3% for CameraCtrl and 30.3% for SVD, with FVD 55.10 vs 105.41 for CameraCtrl.
- On 2-view RealEstate10K generation, Cavia achieves FID 8.82 and FVD 94.86, outperforming CameraCtrl (12.57 / 131.32).
- Ablations show removing cross-frame attention raises FVD on Real10K from 94.86 to 154.54, and removing Plücker conditioning raises it to 195.84.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Cavia instead of CameraCtrl | monocular video generation · RealEstate10K test | FVD: 105.41 -> 55.10 | Helped |
| Use Cavia instead of CameraCtrl | 2-view generation · Real10K | Prec.: 19.49 -> 29.39 | Helped |
| Use Cavia instead of CameraCtrl | 2-view generation · General | MS.: 7.35 -> 19.96 | Helped |
| Add Plücker-coordinate conditioning instead of one-dimensional conditioning (ablation) | Cavia · 2-view Real10K | FID: 12.75 -> 8.82 | Helped |
| Add cross-frame attention instead of vanilla 1D temporal attention (ablation) | Cavia · 2-view Real10K | FID: 17.04 -> 8.82 | Helped |
| Add cross-frame attention instead of vanilla 1D temporal attention (ablation) | Cavia · 2-view General | FID: 71.39 -> 26.12 | Helped |
| Add cross-view attention instead of vanilla spatial attention (ablation) | Cavia · 2-view Real10K | FID: 9.45 -> 8.82 | Helped |
| Add cross-view attention instead of vanilla spatial attention (ablation) | Cavia · 2-view General | FID: 30.89 -> 26.12 | Helped |
| Train on all available data sources rather than static videos only (ablation) | Cavia · 2-view General | FID: 27.20 -> 26.12 | Helped |
| Add monocular video training to synthetic multi-view fine-tuning (ablation) | Cavia · 2-view General | FID: 35.79 -> 26.12 | Helped |
Surprising findings
- Removing monocular video joint training causes generated results to simplify backgrounds when object motion is present, indicating overfitting to synthetic single-color-background Objaverse renders.
Limitations
- Limited ability to generate large object motion, bounded by the SVD base model.
- Assumes a shared camera intrinsic across frames, so complex intrinsic changes are not supported.
- Trained with normalized scene scales rather than metric scale.
Experimental setup
Datasets: RealEstate10K WildRGBD MVImgNet CO3Dv2 DL3DV-10K Objaverse Objaverse-XL InternVid OpenVid Panda-70M
Benchmarks: RealEstate10K
Metrics: FID FVD COLMAP error rotation AUC translation AUC SuperGlue precision SuperGlue matching score
Baselines: SVD MotionCtrl CameraCtrl CVD Vivid-ZOO
Builds on: Stable Video Diffusion MVDream CameraCtrl EDM preconditioning
Closest papers by meaning
- Collaborative Video Diffusion: Consistent Multi-video Generation with Camera Control (NeurIPS 2024 · similarity 0.87)
- GEN3C: 3D-Informed World-Consistent Video Generation with Precise Camera Control (CVPR 2025 · similarity 0.87)
- SynCamMaster: Synchronizing Multi-Camera Video Generation from Diverse Viewpoints (ICLR 2025 · similarity 0.86)
- Trajectory attention for fine-grained video motion control (ICLR 2025 · similarity 0.86)
- Frame In-N-Out: Unbounded Controllable Image-to-Video Generation (NeurIPS 2025 · similarity 0.86)
- AC3D: Analyzing and Improving 3D Camera Control in Video Diffusion Transformers (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.