Research map / Video / Video generation

Training-free Camera Control for Video Generation

ICLR 2025 · Video › Video generation · approach: Efficient video diffusion models · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes CamTrol, a training-free camera control method for off-the-shelf video diffusion models that renders camera-moved image sequences from a point cloud and uses their noisy latents (via inversion) as a layout prior for generation.

Problem

Camera motion control for video diffusion models normally requires finetuning on camera-annotated datasets (e.g., RealEstate10k) or self-supervised augmentation, which is data-hungry and generalizes poorly to other scenes.

Approach

Model camera motion explicitly as rendered views of a single-image point cloud (ZoeDepth + inpainting), then invert those rendered frames into noisy latents and denoise with a frozen video diffusion model so the noise layout prior carries the camera motion into the video.

Key results

• On SVD, CamTrol reaches FVD 778.46 vs 810.59 for MotionCtrl+SVD and 951.80 for CameraCtrl+SVD, with ATE 4.17 (best) and RPE-T 1.07 tied for best.
• Computational analysis shows CamTrol uses the same peak GPU memory as SVD (11542 MB at 576x320) versus 31702 MB for MotionCtrl and 26208 MB for CameraCtrl, and inference takes 8 s vs 32 s and 42 s, with a 56 s preprocessing cost.
• Lowering t0 from 20 to 10 improves FVD from 1079.88 to 754.14 and ATE from 4.17 to 4.13 on SVD, illustrating a fidelity-diversity trade-off.

Measured effects reported in the paper

Surprising findings

• Prompting the base video model with camera-motion text (e.g., 'zooms out') rarely yields the intended motion, while the training-free method does.
• Finetuned camera-control methods (MotionCtrl, CameraCtrl) lose dynamics and fail to generalize to non-realistic styles and scenes with people.

Limitations

• Sequential rendering of multi-view images is a major time cost; parallelization is not explored.
• Large camera motions with few generated frames can cause motion blur artifacts.
• Point cloud from a single image is sparse and rendered frames are static, requiring layout-prior guidance to add dynamics.

Experimental setup

Datasets: RealEstate10k

Benchmarks: RealEstate10k

Metrics: FVD FID IS CLIP-SIM ATE RPE-T RPE-R

Baselines: MotionCtrl CameraCtrl SVD

Builds on: Stable Video Diffusion ZoeDepth Stable Diffusion inpainting DDIM

Closest papers by meaning

• AC3D: Analyzing and Improving 3D Camera Control in Video Diffusion Transformers (CVPR 2025 · similarity 0.92)
• VD3D: Taming Large Video Diffusion Transformers for 3D Camera Control (ICLR 2025 · similarity 0.91)
• Wan-Move: Motion-controllable Video Generation via Latent Trajectory Guidance (NeurIPS 2025 · similarity 0.90)
• Ctrl-Adapter: An Efficient and Versatile Framework for Adapting Diverse Controls to Any Diffusion Model (ICLR 2025 · similarity 0.89)
• Motion-Zero: A Zero-Shot Trajectory Control Framework of Moving Object for Diffusion-Based Video Generation (AAAI 2025 · similarity 0.89)
• ReCapture: Generative Video Camera Controls for User-Provided Videos using Masked Video Fine-Tuning (CVPR 2025 · similarity 0.88)