Research map / 3D vision / 3D generation
CAT4D: Create Anything in 4D with Multi-View Video Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
CAT4D turns a monocular video into a multi-view video with a camera- and time-conditioned multi-view video diffusion model, then reconstructs a dynamic scene as deformable 3D Gaussians.
Problem
Creating 4D (dynamic 3D) scenes from monocular video or generated video is hard because synchronized multi-view video data is scarce, so reconstructions show artifacts from novel viewpoints.
Approach
A multi-view video diffusion model built on CAT3D that separately conditions on camera poses and timestamps (with independent classifier-free guidance), trained on a curated mix of static multi-view, static-view video, synthetic 4D, and generatively augmented data, plus an alternating multi-view/temporal SDEdit sampling strategy that produces unbounded consistent multi-view videos.
Key results
- On NSFF disentangled control, CAT4D reaches 21.97 PSNR for fixed-viewpoint varying-time versus 19.77 for 4DiM.
- On DyCheck 4D reconstruction, CAT4D achieves 17.39 mPSNR versus 16.54 for 4D-GS and 16.72 for Shape-of-Motion, without extra supervision signals.
- On NSFF sparse-view bullet-time reconstruction, CAT4D reaches 20.79 PSNR, above CAT3D-3cond (20.19) and CAT3D-1cond (15.33).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use CAT4D rather than 4DiM | novel-view and time synthesis · NSFF, fixed viewpoint varying time | PSNR: 19.77 -> 21.97 | Helped |
| use CAT4D rather than 4DiM | novel-view and time synthesis · NSFF, varying viewpoint fixed time | PSNR: 18.81 -> 21.68 | Helped |
| use CAT4D rather than 4DiM | novel-view and time synthesis · NSFF, varying viewpoint varying time | PSNR: 17.28 -> 19.73 | Helped |
| use CAT4D rather than CAT3D-3cond | bullet-time reconstruction · NSFF | PSNR: 20.19 -> 20.79 | Helped |
| use CAT4D rather than MoSca | 4D reconstruction · DyCheck | mPSNR: 19.54 -> 17.39 | Hurt |
| use CAT4D rather than MoSca | 4D reconstruction · DyCheck | mLPIPS: 0.244 -> 0.341 | Hurt |
| use independent multi-view sampling without sliding-window overlap instead of alternating sampling (ablation) | CAT4D sampling strategy · NSFF | PSNR: 22.15 -> 20.27 | Hurt |
| use independent temporal sampling without sliding-window overlap instead of alternating sampling (ablation) | CAT4D sampling strategy · NSFF | PSNR: 22.15 -> 21.63 | No clear effect |
| use multi-view sampling only instead of alternating sampling (ablation) | CAT4D sampling strategy · NSFF | PSNR: 22.15 -> 22.34 | Helped |
| use temporal sampling only instead of alternating sampling (ablation) | CAT4D sampling strategy · NSFF | PSNR: 22.15 -> 23.36 | Helped |
Surprising findings
- Alternating sampling has a lower PSNR (22.15) than pure temporal sampling (23.36) on NSFF, though it achieves the best LPIPS (0.108).
Limitations
- Diffusion model struggles with temporal extrapolation beyond input frames and cannot fully disentangle camera viewpoint from time, especially with occlusions.
- Recovered 3D motion fields may not be physically accurate.
- Trained to generate only 16 frames at once due to resource constraints.
Experimental setup
Datasets: CO3D MVImgNet RealEstate10K MC4K Kubric Objaverse (filtered animated assets) static-view monocular videos NSFF DyCheck
Benchmarks: NSFF DyCheck
Metrics: PSNR SSIM LPIPS mPSNR mSSIM mLPIPS
Baselines: 4DiM CAT3D-1cond CAT3D-3cond 4D-GS Shape-of-Motion MoSca
Builds on: CAT3D 4D-GS SDEdit Lumiere data augmentation (SimVS) K-Planes
Closest papers by meaning
- SV4D: Dynamic 3D Content Generation with Multi-Frame and Multi-View Consistency (ICLR 2025 · similarity 0.91)
- 4Diffusion: Multi-view Video Diffusion Model for 4D Generation (NeurIPS 2024 · similarity 0.91)
- CAT3D: Create Anything in 3D with Multi-View Diffusion Models (NeurIPS 2024 · similarity 0.91)
- Controlling Space and Time with Diffusion Models (ICLR 2025 · similarity 0.90)
- Reconstruct, Inpaint, Test-Time Finetune: Dynamic Novel-view Synthesis from Monocular Videos (NeurIPS 2025 · similarity 0.90)
- 4Real: Towards Photorealistic 4D Scene Generation via Video Diffusion Models (NeurIPS 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.