Research map / 3D vision / 3D generation
GEN3C: 3D-Informed World-Consistent Video Generation with Precise Camera Control
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
GEN3C fine-tunes a video diffusion model to condition on renderings of a spatiotemporal 3D cache (depth-unprojected point clouds) so video generation follows precise user camera trajectories and stays 3D-consistent.
Problem
Video generation models lack precise camera control and temporal 3D consistency; objects pop in and out when the camera revisits regions, and sparse-view and dynamic novel view synthesis remain hard.
Approach
Build a 3D cache of colored point clouds from predicted depth of seed images or previously generated frames, render it along the user's camera trajectory, and use the renderings (masked, max-pooled across views, and concatenated to latents) as conditioning for a fine-tuned video diffusion model, so the model only fills disoccluded regions and advances the scene rather than inferring geometry from
Key results
- Single-view video generation on RE10K: PSNR 19.88, LPIPS 0.20, TSED 0.9143, outperforming CameraCtrl (18.40 PSNR, TSED 0.8033).
- Two-view NVS on Tanks-and-Temples extrapolation: PSNR 20.51 / SSIM 0.72 vs MVSplat 16.08 / 0.63.
- Driving NVS FID at y±4.0m: 35.33 vs 3D-GS 81.26 and Nerfacto 112.40 (lower is better).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Condition video generation on renderings of an explicit 3D cache | GEN3C versus CameraCtrl · Tanks-and-Temples single-view video generation | PSNR: 15.88 -> 18.66 | Helped |
| Condition video generation on renderings of an explicit 3D cache | GEN3C versus CameraCtrl · RE10K single-view video generation | PSNR: 18.40 -> 19.88 | Helped |
| Use a rendered 3D cache as guidance for two-view novel view synthesis | GEN3C versus MVSplat · Tanks-and-Temples extrapolation | PSNR: 16.08 -> 20.51 | Helped |
| Use a rendered 3D cache as guidance for two-view novel view synthesis | GEN3C versus MVSplat · RE10K extrapolation | PSNR: 15.51 -> 21.56 | Helped |
| Use GEN3C instead of scene-reconstruction renderers | Driving scene NVS · Driving scene NVS (y ± 4.0m) | FID: 112.40 -> 35.33 | Helped |
| Use GEN3C instead of GCD | Monocular dynamic NVS · Kubric test scenes | FID: 150.64 -> 98.58 | Helped |
Surprising findings
- Plücker-embedding camera-conditioned methods generalize poorly out-of-domain, while GEN3C loses only a small amount of performance on Tanks-and-Temples.
Limitations
- Dynamic content relies on a pre-generated video to supply object motion; text conditioning for motion is not included.
- Depends on depth estimation quality and alignment; fusion of inconsistent multi-view caches can introduce artifacts.
Experimental setup
Datasets: RE10K DL3DV Waymo Open Dataset Kubric4D Tanks-and-Temples
Benchmarks: RE10K Tanks-and-Temples Waymo Open Dataset driving NVS Kubric4D monocular dynamic NVS
Metrics: PSNR SSIM LPIPS TSED FID
Baselines: MotionCtrl CameraCtrl GenWarp NVS-Solver PixelSplat MVSplat Nerfacto 3D-GS GCD
Builds on: Stable Video Diffusion Depth Anything V2 DROID-SLAM latent diffusion models
Closest papers by meaning
- GenFusion: Closing the Loop between Reconstruction and Generation via Videos (CVPR 2025 · similarity 0.88)
- AC3D: Analyzing and Improving 3D Camera Control in Video Diffusion Transformers (CVPR 2025 · similarity 0.88)
- VD3D: Taming Large Video Diffusion Transformers for 3D Camera Control (ICLR 2025 · similarity 0.88)
- GenXD: Generating Any 3D and 4D Scenes (ICLR 2025 · similarity 0.87)
- SimVS: Simulating World Inconsistencies for Robust View Synthesis (CVPR 2025 · similarity 0.87)
- Wan-Move: Motion-controllable Video Generation via Latent Trajectory Guidance (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.