Research map / 3D vision / 3D generation
GenXD: Generating Any 3D and 4D Scenes
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces CamVid-30K, a 30K-video real-world 4D scene dataset curated with SfM camera poses and object motion strength, and GenXD, a latent diffusion model with multiview-temporal modules and mask-latent conditioning that generates 3D and 4D scenes from any number of input views.
Problem
General 3D and 4D (dynamic 3D) scene generation is hard because real-world 4D data with both camera and object movement is scarce, and existing models handle only object-centric synthetic data or fixed conditioning views.
Approach
Curate camera poses (mask-based SfM) and a depth-aligned object motion field from videos to build a 4D scene dataset, then train one SVD-initialized diffusion model with separate multiview and temporal layers fused by a learnable alpha (set to 0 for 3D data) and mask-latent conditioning so any number of condition views can be used.
Key results
- 4D scene generation on Cam-DAVIS: GenXD (3 views) reaches FID 55.64 and FVD 490.50 vs. MotionCtrl 118.14/1464.08 and CameraCtrl 138.64/1470.59.
- Few-view 3D reconstruction on Re10K: Zip-NeRF + GenXD raises PSNR from 20.58 to 25.40; 3D-GS + GenXD raises PSNR from 18.84 to 23.13.
- 4D object generation: GenXD (single view) achieves CLIP-I 90.32 in about 4 minutes versus Animate124's 85.44 in 7 hours.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use GenXD with 3 conditioning views instead of MotionCtrl | 4D scene generation · Cam-DAVIS | FID: 118.14 -> 55.64 | Helped |
| Use GenXD with 3 conditioning views instead of MotionCtrl | 4D scene generation · Cam-DAVIS | FVD: 1464.08 -> 490.50 | Helped |
| Use GenXD instead of Animate124 | 4D object generation · 4D object generation | CLIP-I: 85.44 -> 90.32 | Helped |
| Use GenXD video generation and 4D-GS instead of Animate124 optimization | 4D object generation · 4D object generation | Time: 7 hrs -> 4 min | Helped |
| Add generated views from GenXD to Zip-NeRF | Zip-NeRF · Re10K | PSNR: 20.58 -> 25.40 | Helped |
| Add generated views from GenXD to Zip-NeRF | Zip-NeRF · LLFF | PSNR: 14.26 -> 19.39 | Helped |
| Add motion disentanglement (ablation) | GenXD · Re10K | PSNR: 20.75 -> 22.96 | Helped |
| Add motion disentanglement (ablation) | GenXD · Cam-DAVIS | FID: 122.73 -> 101.78 | Helped |
| Use Plücker-ray camera conditioning instead of Camera CA (ablation) | GenXD · Re10K | PSNR: 21.73 -> 22.96 | Helped |
| Add 3D training data (ablation) | GenXD · Re10K | PSNR: 16.38 -> 22.96 | Helped |
| Add 4D training data (ablation) | GenXD · Re10K | PSNR: 20.74 -> 22.96 | Helped |
| Use GenXD instead of IM-3D | single-view 3D generation · single-view 3D generation | CLIP-I: 91.40 -> 84.75 | Hurt |
Surprising findings
- Removing 3D data severely hurts 4D-related and 3D reconstruction quality (Re10K PSNR drops from 22.96 to 16.38 in the ablation), showing 3D data is important for both settings.
Limitations
- Struggles to generate 360-degree views of complex scenes from single-view conditions.
- Temporal consistency and object motion are hard to maintain under large camera movements due to limited available video data.
- Real-world 3D datasets have limited diversity, often forward-facing camera trajectories.
Experimental setup
Datasets: Objaverse MVImageNet Co3D RealEstate10K ACID Objaverse-XL-Animation CamVid-30K DAVIS LLFF
Benchmarks: Cam-DAVIS Re10K LLFF CLIP-I image-to-3D examples
Metrics: FID FVD PSNR SSIM LPIPS CLIP-I generation time
Baselines: MotionCtrl CameraCtrl Animate124 Zero-1-to-3-V RealFusion-V IM-3D ImageDream One2345++
Builds on: Stable Video Diffusion Particle-SfM latent diffusion model 3D Gaussian Splatting Zip-NeRF
Closest papers by meaning
- DreamScene4D: Dynamic Multi-Object Scene Generation from Monocular Videos (NeurIPS 2024 · similarity 0.90)
- SV4D: Dynamic 3D Content Generation with Multi-Frame and Multi-View Consistency (ICLR 2025 · similarity 0.89)
- 4Real: Towards Photorealistic 4D Scene Generation via Video Diffusion Models (NeurIPS 2024 · similarity 0.88)
- EG4D: Explicit Generation of 4D Object without Score Distillation (ICLR 2025 · similarity 0.88)
- GEN3C: 3D-Informed World-Consistent Video Generation with Precise Camera Control (CVPR 2025 · similarity 0.87)
- Robust Multi-Object 4D Generation for In-the-wild Videos (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.