Research map / 3D vision / 3D generation
DreamComposer: Controllable 3D Object Generation via Multi-View Conditions
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DreamComposer, a plug-in framework that lifts multiple input views into tri-plane latent 3D features, renders target-view features, and injects them via a ControlNet-style branch into view-conditioned diffusion models (Zero-1-to-3, SyncDreamer) for controllable novel view synthesis.
Problem
Single-view-conditioned novel view synthesis and 3D object generation from diffusion models are not fully controllable because a single input view lacks information about unseen parts of the object.
Approach
A target-aware 3D lifting module (tri-plane latents conditioned on relative angle via cross-attention), weighted composited volume rendering of multi-view features into the target view, and ControlNet-style injection into a frozen pre-trained view-conditioned diffusion UNet, allowing arbitrary numbers of input views.
Key results
- On GSO with multi-view inputs, Zero-1-to-3+DreamComposer reaches PSNR 25.25 vs 20.82 for Zero-1-to-3 at 0-degree elevation.
- SyncDreamer+DreamComposer improves PSNR from 20.05 to 20.52 and LPIPS from 0.146 to 0.141 on GSO single-view setting.
- Removing view conditioning in 3D lifting drops PSNR to 19.04 and fine-tuning the UNet drops it to 15.96 in ablations.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add multi-view conditions to Zero-1-to-3 | Zero-1-to-3 · GSO, elevations 0, 15, and 30 | PSNR: 20.82 -> 25.25; 21.38 -> 25.85; 21.66 -> 25.63 | Helped |
| add multi-view conditions to Zero-1-to-3 | Zero-1-to-3 · GSO, elevations 0, 15, and 30 | SSIM: 0.840 -> 0.888; 0.837 -> 0.891; 0.837 -> 0.885 | Helped |
| add multi-view conditions to Zero-1-to-3 | Zero-1-to-3 · GSO, elevations 0, 15, and 30 | LPIPS: 0.139 -> 0.088; 0.131 -> 0.083; 0.128 -> 0.086 | Helped |
| add multi-view conditions | SyncDreamer · GSO | PSNR: 20.05 -> 20.52 | Helped |
| add multi-view conditions | SyncDreamer · GSO | SSIM: 0.798 -> 0.828 | Helped |
| add multi-view conditions | SyncDreamer · GSO | LPIPS: 0.146 -> 0.141 | Helped |
| freeze the diffusion U-Net instead of finetuning it (ablation) | DreamComposer with SyncDreamer · GSO | PSNR: 15.96 -> 20.52 | Helped |
| add reconstruction MSE loss (ablation) | DreamComposer with SyncDreamer · GSO | PSNR: 16.18 -> 20.52 | Helped |
| add view conditioning to 3D lifting (ablation) | DreamComposer with SyncDreamer · GSO | PSNR: 19.04 -> 20.52 | Helped |
Surprising findings
- Fine-tuning the pre-trained diffusion UNet together with the new modules degrades performance (PSNR 15.96) compared with freezing it.
Limitations
- Dependence on a pre-trained view-conditioned diffusion model as frozen backbone
- Views of unseen regions still rely on generated auxiliary views when only one ground-truth input is available
Experimental setup
Datasets: Objaverse Google Scanned Objects (GSO)
Benchmarks: Google Scanned Objects (GSO)
Metrics: PSNR SSIM LPIPS
Baselines: Zero-1-to-3 SyncDreamer RealFusion
Builds on: Zero-1-to-3 SyncDreamer ControlNet Stable Diffusion NeuS
Closest papers by meaning
- DreamControl: Control-Based Text-to-3D Generation with 3D Self-Prior (CVPR 2024 · similarity 0.89)
- MVDream: Multi-view Diffusion for 3D Generation (ICLR 2024 · similarity 0.89)
- DreamScene4D: Dynamic Multi-Object Scene Generation from Monocular Videos (NeurIPS 2024 · similarity 0.88)
- Ouroboros3D: Image-to-3D Generation via 3D-aware Recursive Diffusion (CVPR 2025 · similarity 0.88)
- MagicMan: Generative Novel View Synthesis of Humans with 3D-Aware Diffusion and Iterative Refinement (AAAI 2025 · similarity 0.88)
- UMAMI: Unifying Masked Autoregressive Models and Deterministic Rendering for View Synthesis (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.