Research map / Video / Video generation
AC3D: Analyzing and Improving 3D Camera Control in Video Diffusion Transformers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Analyzes how camera motion is modeled in an 11.5B video DiT and improves 3D camera control via truncated noise schedules, layer-limited camera conditioning, and a 20K stationary-camera dynamic-scene dataset.
Problem
Camera control added to pretrained text-to-video diffusion transformers is imprecise and degrades visual quality and scene dynamics.
Approach
Motion spectral analysis shows camera motion is low-frequency and determined early in denoising, so camera conditioning is restricted to high-noise steps; linear probing shows camera pose is encoded in early-middle blocks, so conditioning is limited to the first 8 blocks; and a 20K dynamic-scene, stationary-camera dataset rebalances training to separate camera from scene motion.
Key results
- AC3D (VDiT) reaches RE10K translation error 0.358 and rotation error 0.035, versus 0.421 and 0.056 for VD3D (VDiT).
- On MSR-VTT, AC3D (VDiT) achieves FVD 110.71 versus 137.62 for VD3D (VDiT), and CLIP score 28.58 versus 27.90.
- User study: AC3D preferred over VD3D (DiT) overall in 92.5% of comparisons and over VD3D (FIT) in 95.0%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use AC3D instead of VD3D | VDiT camera-controlled video generation · MSR-VTT | FVD: 110.71 vs 137.62 | Helped |
| remove camera conditioning (ablation) | AC3D · MSR-VTT | FVD: -8.53 | Helped |
| remove high-noise biasing (ablation) | AC3D · MSR-VTT | FVD: +16.92 | Hurt |
| remove noise truncation (ablation) | AC3D · MSR-VTT | FVD: +6.63 | Hurt |
| remove camera classifier-free guidance (ablation) | AC3D · MSR-VTT | FVD: +1.42 | Hurt |
| remove curated dynamic-scene data (ablation) | AC3D · MSR-VTT | FVD: +4.40 | Hurt |
| use unscaled RealEstate10K camera parameters (ablation) | AC3D · MSR-VTT | FVD: 0.00 | No clear effect |
| provide context to the camera branch (ablation) | AC3D · MSR-VTT | FVD: +7.41 | Hurt |
| extend camera conditioning beyond the first 8 blocks (ablation) | AC3D · MSR-VTT | FVD: +9.23 | Hurt |
| add joint training on 2D video data (ablation) | AC3D · MSR-VTT | FVD: -3.83 | Helped |
Surprising findings
- Restricting camera conditioning to the first 8 of 32 blocks improves visual quality by about 10% rather than hurting it.
- Joint training with 2D data without camera annotations improved FVD slightly but made camera steering up to 3x worse.
Limitations
- Camera control for trajectories far outside the training distribution is not addressed.
- Scale ambiguity and annotation difficulty for dynamic scenes limit camera data; a stationary-camera dataset is used instead.
Experimental setup
Datasets: RealEstate10K MSR-VTT curated dynamic stationary-camera videos
Benchmarks: RealEstate10K MSR-VTT
Metrics: FID FVD CLIP score rotation error translation error
Baselines: MotionCtrl CameraCtrl VD3D CogVideoX
Builds on: VD3D ControlNet rectified flow MotionCtrl CameraCtrl
Closest papers by meaning
- VD3D: Taming Large Video Diffusion Transformers for 3D Camera Control (ICLR 2025 · similarity 0.93)
- Training-free Camera Control for Video Generation (ICLR 2025 · similarity 0.92)
- GEN3C: 3D-Informed World-Consistent Video Generation with Precise Camera Control (CVPR 2025 · similarity 0.88)
- Controlling Space and Time with Diffusion Models (ICLR 2025 · similarity 0.88)
- Boximator: Generating Rich and Controllable Motions for Video Synthesis (ICML 2024 · similarity 0.88)
- FloVD: Optical Flow Meets Video Diffusion Model for Enhanced Camera-Controlled Video Synthesis (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.