Research map / 3D vision / 3D generation
Wonder3D: Single Image to 3D using Cross-Domain Diffusion
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Wonder3D generates consistent multi-view normal maps and color images with a cross-domain diffusion model fine-tuned from Stable Diffusion, then fuses them via a geometry-aware SDF optimization to produce textured meshes from a single image in 2-3 minutes.
Problem
Single-view 3D reconstruction that is high-fidelity, multi-view consistent, generalizable, and efficient, avoiding slow per-shape SDS optimization and low-quality feed-forward 3D generators.
Approach
Jointly model multi-view normal maps and color images with a single diffusion UNet using a domain switcher and cross-domain attention (initialized from Stable Diffusion Image Variations), then reconstruct an SDF using a geometry-aware normal loss with ray-angle weighting and outlier dropping.
Key results
- Chamfer Distance 0.0199 and Volume IoU 0.6244 on GSO, beating SyncDreamer (0.0261, 0.5421) and Zero123 (0.0339, 0.5035).
- Novel view synthesis PSNR 26.07, SSIM 0.924, LPIPS 0.065 on GSO, versus 20.05 PSNR for SyncDreamer.
- Reconstructs textured meshes from a single image in about 2-3 minutes; training takes ~3 days on 8 A800 GPUs for 30k steps.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Wonder3D instead of SyncDreamer for single-image 3D reconstruction | Wonder3D · GSO | Chamfer Distance: 0.0261 -> 0.0199 | Helped |
| Use Wonder3D instead of SyncDreamer for single-image 3D reconstruction | Wonder3D · GSO | Volume IoU: 0.5421 -> 0.6244 | Helped |
| Use Wonder3D instead of SyncDreamer for novel view synthesis | Wonder3D · GSO | PSNR: 20.05 -> 26.07 | Helped |
| Use Wonder3D instead of SyncDreamer for novel view synthesis | Wonder3D · GSO | LPIPS: 0.146 -> 0.065 | Helped |
| Remove cross-domain attention (ablation) | Wonder3D cross-domain diffusion · Generated multi-view normal maps and color images | Cross-domain consistency | Hurt |
| Replace the joint cross-domain model with sequential color-to-normal models (ablation) | Wonder3D cross-domain diffusion · Generated multi-view normal maps and color images | Generation quality | Hurt |
| Replace the joint cross-domain model with sequential normal-to-color models (ablation) | Wonder3D cross-domain diffusion · Generated multi-view normal maps and color images | Generation quality | Hurt |
| Add geometry-aware normal loss (ablation) | Wonder3D normal fusion · Lion model mesh extraction | Surface quality | Helped |
| Add outlier-dropping strategy (ablation) | Wonder3D normal fusion · Lion model mesh extraction | Surface quality | Helped |
| Add both geometry-aware normal loss and outlier-dropping strategy (ablation) | Wonder3D normal fusion · Lion model mesh extraction | Surface quality | Helped |
Surprising findings
- Naive channel expansion of the UNet output for normals performs poorly, attributed to catastrophic forgetting of pretrained weights.
- Sequential color-to-normal or normal-to-color models maintain some consistency but show color shifts or wrong geometry, unlike the joint model.
Limitations
- Only six views generated, making thin structures and severe occlusions hard to reconstruct.
- Relies on segmentation models to obtain object masks.
- Evaluated on a small GSO set of 30 objects.
Experimental setup
Datasets: Objaverse (LVIS subset) Google Scanned Objects (GSO)
Benchmarks: Google Scanned Objects (GSO)
Metrics: Chamfer Distance Volume IoU PSNR SSIM LPIPS
Baselines: Zero123 RealFusion Magic123 One-2-3-45 Point-E Shap-E SyncDreamer
Builds on: Stable Diffusion Zero123 SyncDreamer MVDream instant-NGP NeuS
Closest papers by meaning
- One-2-3-45++: Fast Single Image to 3D Objects with Consistent Multi-View Generation and 3D Diffusion (CVPR 2024 · similarity 0.90)
- Unique3D: High-Quality and Efficient 3D Mesh Generation from a Single Image (NeurIPS 2024 · similarity 0.90)
- Cycle3D: High-quality and Consistent Image-to-3D Generation via Generation-Reconstruction Cycle (AAAI 2025 · similarity 0.89)
- DreamCraft3D: Hierarchical 3D Generation with Bootstrapped Diffusion Prior (ICLR 2024 · similarity 0.88)
- Magic123: One Image to High-Quality 3D Object Generation Using Both 2D and 3D Diffusion Priors (ICLR 2024 · similarity 0.88)
- Era3D: High-Resolution Multiview Diffusion using Efficient Row-wise Attention (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.