Research map / Video / Video generation
VD3D: Taming Large Video Diffusion Transformers for 3D Camera Control
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VD3D, a ControlNet-like conditioning mechanism that injects pixel-wise Plücker camera embeddings into a joint spatiotemporal video diffusion transformer (SnapVideo/FIT) to enable 3D camera control for text-to-video generation, fine-tuned on RealEstate10K.
Problem
Text-to-video diffusion transformers that jointly model space and time lack fine-grained camera pose control; existing camera-control methods rely on temporal layers of U-Net models and do not transfer to them.
Approach
Represent camera trajectories as per-pixel Plücker coordinates patchified into spatiotemporal tokens, and inject them via zero-initialized ControlNet-style read cross-attention branches in each FIT block, while freezing the base model.
Key results
- On RealEstate10K, VD3D reaches TransError 0.409 and RotError 0.043 vs. 0.465/0.089 for the best baseline (CameraCtrl U-Net).
- In a 20-participant user study, VD3D is preferred over MotionCtrl overall 84% and over CameraCtrl overall 66%, with camera alignment preferred 82% and 78%.
- Under a vanilla CogVideoX DiT backbone, VD3D again improves camera accuracy (RealEstate10K TransError 0.421 vs. 0.501 for MotionCtrl).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use VD3D camera conditioning instead of MotionCtrl | SnapVideo · RealEstate10K test | TransError: 0.518 -> 0.409 | Helped |
| Use VD3D camera conditioning instead of MotionCtrl | SnapVideo · MSR-VTT | RotError: 0.148 -> 0.050 | Helped |
| Use VD3D camera conditioning instead of CameraCtrl | SnapVideo · RealEstate10K test | RotError: 0.165 -> 0.043 | Helped |
| Use VD3D camera conditioning instead of CameraCtrl | SnapVideo · MSR-VTT | TransError: 0.578 -> 0.504 | Helped |
| Add Plücker embeddings (ablation) | VD3D · RealEstate10K test | TransError: 0.517 -> 0.409 | Helped |
| Add Plücker embeddings (ablation) | VD3D · MSR-VTT | RotError: 0.156 -> 0.050 | Helped |
| Add ControlNet conditioning (ablation) | VD3D · RealEstate10K test | TransError: 0.573 -> 0.409 | Helped |
| Add ControlNet conditioning (ablation) | VD3D · MSR-VTT | TransError: 0.787 -> 0.504 | Helped |
| Copy pre-trained weights into the new cross-attention block (ablation) | VD3D · RealEstate10K test | TransError: 0.424 -> 0.409 | Helped |
| Copy pre-trained weights into the new cross-attention block (ablation) | VD3D · MSR-VTT | RotError: 0.063 -> 0.050 | Helped |
| Add camera matrices to the context vector (ablation) | VD3D · RealEstate10K test | TransError: 0.602 -> 0.409 | Helped |
| Add camera matrices to the context vector (ablation) | VD3D · MSR-VTT | RotError: 0.128 -> 0.050 | Hurt |
Surprising findings
- Adding camera embeddings to the context vector or injecting them as zero-initialized residuals yields little to no camera control.
- Copying pretrained weights into the ControlNet branch has minor impact; random initialization gives similar results when zero-initialized output convolutions are used.
Limitations
- Dynamic scenes rendered from different viewpoints can show inconsistent motion.
- Only the low-resolution base model is camera-conditioned; the upsampler is frozen.
- Generates short videos (16 frames).
Experimental setup
Datasets: RealEstate10K MSR-VTT
Benchmarks: RealEstate10K MSR-VTT
Metrics: TransError RotError PSNR SSIM LPIPS FID FVD CLIPSIM human preference (camera alignment, motion quality, text ali
Baselines: MotionCtrl CameraCtrl MotionCtrl (U-Net) CameraCtrl (U-Net) Base Model
Builds on: SnapVideo ControlNet MotionCtrl CameraCtrl Far-reaching Interleaved Transformers
Closest papers by meaning
- AC3D: Analyzing and Improving 3D Camera Control in Video Diffusion Transformers (CVPR 2025 · similarity 0.93)
- Training-free Camera Control for Video Generation (ICLR 2025 · similarity 0.91)
- DiTCtrl: Exploring Attention Control in Multi-Modal Diffusion Transformer for Tuning-Free Multi-Prompt Longer Video Generation (CVPR 2025 · similarity 0.90)
- Generative Rendering: Controllable 4D-Guided Video Generation with 2D Diffusion Models (CVPR 2024 · similarity 0.90)
- Ctrl-Adapter: An Efficient and Versatile Framework for Adapting Diverse Controls to Any Diffusion Model (ICLR 2025 · similarity 0.89)
- Snap Video: Scaled Spatiotemporal Transformers for Text-to-Video Synthesis (CVPR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.