Research map / 3D vision / 3D generation
CAT3D: Create Anything in 3D with Multi-View Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
CAT3D generates many consistent novel views with a multi-view latent diffusion model (initialized from a text-to-image LDM, using 3D self-attention and raymap conditioning) and then reconstructs a NeRF from them with a robust Zip-NeRF variant with LPIPS and distance-weighted losses.
Problem
Creating 3D scenes from few or single input images is ill-posed for standard reconstruction because of too few supervising views, while video or per-view generative priors are costly, hard to control, or lack 3D consistency.
Approach
Reformulate sparse-view 3D creation as generation: a multi-view diffusion model (text-to-image LDM inflated with 3D self-attention and raymap camera conditioning, conditioned on any number of input views) produces dozens to hundreds of novel views that are then fused by a robust NeRF with LPIPS loss and distance-based view weighting.
Key results
- On 3-view RealEstate10K, CAT3D reports 26.78 PSNR vs 25.84 for ReconFusion; on 3-view mip-NeRF 360 it reports 16.62 PSNR vs 15.50 for ReconFusion.
- CAT3D achieves state-of-the-art on nearly all few-view settings (3/6/9 views across five datasets) while cutting generation time from about 1 hour (ZeroNVS, ReconFusion) to a few minutes.
- Single-image CAT3D gets CLIP image score 88.54 in about 1 minute, versus 83.77 for ImageDream (120 min) and 91.40 for IM-3D (3 min).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use CAT3D instead of ReconFusion | Few-view 3D reconstruction · RealEstate10K 3-view | PSNR: 25.84 -> 26.78 | Helped |
| Use CAT3D instead of ReconFusion | Few-view 3D reconstruction · LLFF 3-view | PSNR: 21.34 -> 21.58 | Helped |
| Use CAT3D instead of ReconFusion | Few-view 3D reconstruction · DTU 3-view | PSNR: 20.74 -> 22.02 | Helped |
| Use CAT3D instead of ReconFusion | Few-view 3D reconstruction · CO3D 3-view | PSNR: 19.59 -> 20.57 | Helped |
| Use CAT3D instead of ReconFusion | Few-view 3D reconstruction · Mip-NeRF 360 3-view | PSNR: 15.50 -> 16.62 | Helped |
| Generate 5 target views rather than 1 (ablation) | CAT3D multi-view diffusion model · In-domain diffusion samples | PSNR: 18.85 -> 21.66 | Helped |
| Add raymap camera conditioning instead of a low-dimensional vector (ablation) | CAT3D multi-view diffusion model · Out-of-domain diffusion samples | PSNR: 14.19 -> 14.63 | Helped |
| Use 3D self-attention instead of temporal attention (ablation) | CAT3D multi-view diffusion model · In-domain diffusion samples | PSNR: 18.62 -> 21.66 | Helped |
| Extend 3D self-attention from 32 × 32 to 64 × 64 feature maps (ablation) | CAT3D multi-view diffusion model · In-domain diffusion samples | PSNR: 21.66 -> 22.83 | Helped |
| Extend 3D self-attention from 32 × 32 to 64 × 64 feature maps (ablation) | CAT3D multi-view diffusion model · Out-of-domain diffusion samples | PSNR: 14.63 -> 14.64 | No clear effect |
| Initialize from a pretrained text-to-image LDM rather than training from scratch (ablation) | CAT3D multi-view diffusion model · Out-of-domain diffusion samples | PSNR: 13.88 -> 14.63 | Helped |
| Train the pretrained model for 1M iterations rather than the shorter training setting (ablation) | CAT3D multi-view diffusion model · Out-of-domain diffusion samples | PSNR: 14.63 -> 15.19 | Helped |
Surprising findings
- Replacing PixelNeRF feature conditioning and CLIP image embeddings with 3D self-attention conditioning removes the need for them while improving results.
- Increasing generated views from 80 to 720 improved central-object geometry but sometimes added background blur from generated inconsistencies.
Limitations
- Trained data assumes roughly constant camera intrinsics per scene, so it fails with mixed-camera inputs.
- Generation quality depends on the base text-to-image model and degrades for out-of-distribution content.
- Camera trajectories are manually designed, and generated views are not perfectly 3D consistent.
Experimental setup
Datasets: Objaverse CO3D RealEstate10K MVImgNet DTU LLFF mip-NeRF 360
Benchmarks: RealEstate10K LLFF DTU CO3D mip-NeRF 360
Metrics: PSNR SSIM LPIPS CLIP image score
Baselines: Zip-NeRF ZeroNVS ReconFusion ImageDream One2345++ IM-3D DreamCraft3D RealmDreamer
Builds on: Zip-NeRF latent diffusion model ReconFusion ImageDream FlashAttention
Closest papers by meaning
- CAT4D: Create Anything in 4D with Multi-View Video Diffusion Models (CVPR 2025 · similarity 0.91)
- Sculpt3D: Multi-View Consistent Text-to-3D Generation with Sparse 3D Prior (CVPR 2024 · similarity 0.89)
- One-2-3-45++: Fast Single Image to 3D Objects with Consistent Multi-View Generation and 3D Diffusion (CVPR 2024 · similarity 0.89)
- Denoising Diffusion via Image-Based Rendering (ICLR 2024 · similarity 0.89)
- Efficient-3Dim: Learning a Generalizable Single-image Novel-view Synthesizer in One Day (ICLR 2024 · similarity 0.88)
- Cycle3D: High-quality and Consistent Image-to-3D Generation via Generation-Reconstruction Cycle (AAAI 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.