Research map / Video / Video generation
SynCamMaster: Synchronizing Multi-Camera Video Generation from Diverse Viewpoints
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Adds a plug-and-play multi-view synchronization module with camera encoder to a frozen internal DiT text-to-video model, enabling synchronized multi-camera video generation from arbitrary 6-DoF viewpoints, trained with a hybrid of UE-rendered multi-view videos, DL3DV multi-view images, and filtered monocular videos.
Problem
Generating synchronized multi-camera videos of the same open-domain dynamic scene from arbitrary viewpoints is under-explored; prior multi-view video work targets single-object 4D assets from fixed orbit positions, and real-world multi-view video data is scarce.
Approach
Insert a cross-view self-attention module and a per-camera extrinsic encoder into each DiT block of a frozen pre-trained T2V model, and train it with a hybrid data scheme (multi-view images from single-camera videos, UE-rendered multi-camera videos, and static-camera monocular videos replicated as overlapping views) with progressively increasing viewpoint angle differences.
Key results
- SynCamMaster reaches FVD-V 1470 and CLIP-V 93.71 for cross-view synchronization versus 1742/89.14 for M.V. Image + SVD-XT and 1930/90.01 for M.V. Image + I2V-Ours.
- Camera control error on 100 videos drops to RotErr 0.12 and TransErr 0.58, compared with 0.16/0.67 for CameraCtrl.
- Joint training with multi-view images and general video raises FID-style fidelity from 149.9 FID (multi-view video only) to 116.7 FID and FVD from 1971 to 1401.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add multi-view synchronization and camera conditioning | SynCamMaster · Multi-camera video generation | Mat. Pix. (K): 116.8 -> 527.1 | Helped |
| Add multi-view synchronization and camera conditioning | SynCamMaster · Multi-camera video generation | FVD-V: 1930 -> 1470 | Helped |
| Add multi-view synchronization and camera conditioning | SynCamMaster · Multi-camera video generation | CLIP-V: 90.01 -> 93.71 | Helped |
| Add multi-view image data to multi-view video training (ablation) | SynCamMaster · Joint-training ablation | FID: 149.9 -> 121.5 | Helped |
| Add multi-view image data to multi-view video training (ablation) | SynCamMaster · Joint-training ablation | Mat. Pix. (K): 460.5 -> 533.0 | Helped |
| Add general video data to multi-view video training (ablation) | SynCamMaster · Joint-training ablation | FID: 149.9 -> 122.4 | Helped |
| Add general video data to multi-view video training (ablation) | SynCamMaster · Joint-training ablation | FVD: 1971 -> 1608 | Helped |
| Add both multi-view image and general video data (ablation) | SynCamMaster · Joint-training ablation | FID: 149.9 -> 116.7 | Helped |
| Add both multi-view image and general video data (ablation) | SynCamMaster · Joint-training ablation | CLIP-V: 89.68 -> 93.71 | Helped |
| Use progressive training with gradually larger viewpoint differences (ablation) | SynCamMaster · Large viewpoint differences | camera pose following | Helped |
| Replace full attention with epipolar attention (ablation) | SynCamMaster view attention · Camera control | RotErr: 0.12 -> 0.10 | Helped |
| Replace full attention with epipolar attention (ablation) | SynCamMaster view attention · Camera control | TransErr: 0.58 -> 0.59 | Hurt |
Surprising findings
- Epipolar attention gives lower rotation error (0.10 vs 0.12) but degrades text-prompt alignment, so full attention was preferred.
- Using too much single-view video data biases the model toward small relative angles and hurts camera accuracy.
Limitations
- Inconsistent details in complex scenes such as bowls and plates across views.
- Inherits base model weaknesses such as poor hand generation.
- Base T2V model is internal and not publicly available, limiting reproducibility.
Experimental setup
Datasets: DL3DV-10K RealEstate-10K Unreal Engine rendered multi-view videos filtered general monocular videos
Benchmarks: VBench
Metrics: FID FVD CLIP-T CLIP-F Mat. Pix. FVD-V CLIP-V RotErr TransErr LPIPS
Baselines: SVD-XT + multi-view image CameraCtrl I2V-Ours GCD
Builds on: CameraCtrl CVD Rectified Flow Instruct-Pix2Pix EMU Video
Closest papers by meaning
- SyncDreamer: Generating Multiview-consistent Images from a Single-view Image (ICLR 2024 · similarity 0.89)
- 4Real-Video: Learning Generalizable Photo-Realistic 4D Video Diffusion (CVPR 2025 · similarity 0.89)
- SimVS: Simulating World Inconsistencies for Robust View Synthesis (CVPR 2025 · similarity 0.89)
- Collaborative Video Diffusion: Consistent Multi-video Generation with Camera Control (NeurIPS 2024 · similarity 0.88)
- Vivid-ZOO: Multi-View Video Generation with Diffusion Model (NeurIPS 2024 · similarity 0.88)
- SV4D: Dynamic 3D Content Generation with Multi-Frame and Multi-View Consistency (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.