Research map / 3D vision / 3D generation
Magic123: One Image to High-Quality 3D Object Generation Using Both 2D and 3D Diffusion Priors
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Magic123 generates high-quality textured 3D meshes from a single image via a two-stage coarse-to-fine pipeline (Instant-NGP NeRF then DMTet mesh) that jointly uses 2D (Stable Diffusion) and 3D (Zero-1-to-3) diffusion priors via score distillation.
Problem
Single-image 3D reconstruction in the wild is ill-posed; 2D diffusion priors generalize well but yield Janus artifacts and inconsistent geometry, while 3D priors are accurate for common objects but oversimplify uncommon ones due to limited 3D data.
Approach
Combine a 2D diffusion prior (Stable Diffusion with textual inversion) and a view-conditioned 3D prior (Zero-1-to-3) in one SDS loss with a single trade-off weight lambda_2D/3D, applied in a NeRF coarse stage and a DMTet high-resolution mesh fine stage, plus depth and normal regularization.
Key results
- On NeRF4, Magic123 reaches CLIP-similarity 0.80, PSNR 24.62, LPIPS 0.03, beating Zero-1-to-3 (0.62 / 23.96 / 0.05).
- On RealFusion15, Magic123 reports CLIP-similarity 0.82 and LPIPS 0.10, the best among compared methods, with PSNR 19.50 vs 19.49 for Zero-1-to-3.
- Using a fixed setting (lambda_2D/3D=1, lambda_3D=40) across all examples, the 3D-only prior is robust to lambda_3D in 10-60, while the 2D prior is sensitive to lambda_2D.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Magic123 instead of Zero-1-to-3 | Image-to-3D reconstruction · NeRF4 | CLIP-Similarity: 0.62 -> 0.80 | Helped |
| Use Magic123 instead of Zero-1-to-3 | Image-to-3D reconstruction · NeRF4 | PSNR: 23.96 -> 24.62 | Helped |
| Use Magic123 instead of Zero-1-to-3 | Image-to-3D reconstruction · NeRF4 | LPIPS: 0.05 -> 0.03 | Helped |
| Use Magic123 instead of Zero-1-to-3 | Image-to-3D reconstruction · RealFusion15 | CLIP-Similarity: 0.75 -> 0.82 | Helped |
| Use Magic123 instead of Zero-1-to-3 | Image-to-3D reconstruction · RealFusion15 | PSNR: 19.49 -> 19.50 | Helped |
| Use Magic123 instead of Zero-1-to-3 | Image-to-3D reconstruction · RealFusion15 | LPIPS: 0.11 -> 0.10 | Helped |
| Add the fine DMTet mesh stage after the coarse stage (ablation) | Magic123 · Image-to-3D reconstruction | quantitative performance | Helped |
| Use only the 3D prior and vary λ3D from 10 to 80 (ablation) | Magic123 · NeRF4 | CLIP-similarity: 0.58, 0.61, 0.62, 0.61, 0.58 | No clear effect |
| Increase λ2D from 0.1 to 2 using only the 2D prior (ablation) | Magic123 · NeRF4 | CLIP-similarity: 0.54 -> 0.72 | Helped |
| Increase λ2D from 0.1 to 2 using only the 2D prior (ablation) | Magic123 · NeRF4 | PSNR: 23.62 -> 22.42 | Hurt |
| Increase λ2D from 0.1 to 2 using only the 2D prior (ablation) | Magic123 · NeRF4 | LPIPS: 0.04 -> 0.07 | Hurt |
| Rely solely on the 3D prior instead of combining 2D and 3D priors (ablation) | Magic123 · Uncommon objects such as dragon statues | geometry detail | Hurt |
Surprising findings
- Zero-1-to-3 as a 3D prior is much more tolerant to its guidance weight than Stable Diffusion is to its own, so the trade-off is tuned mainly through the 2D weight.
Limitations
- Assumes the reference image is taken from a front view; poor geometry otherwise.
- Depends on off-the-shelf segmentation and monocular depth estimation models, whose errors propagate.
- Over-saturated textures due to SDS, more severe at the high-resolution fine stage.
Experimental setup
Datasets: NeRF synthetic (4 scenes: chair, drums, ficus, microphone) RealFusion15 Objaverse (used by Zero-1-to-3 prior)
Benchmarks: NeRF4 RealFusion15
Metrics: CLIP-similarity PSNR LPIPS
Baselines: Point-E Shap-E 3DFuse NeuralLift RealFusion Zero-1-to-3
Builds on: DreamFusion Zero-1-to-3 RealFusion Magic3D DMTet
Closest papers by meaning
- One-2-3-45++: Fast Single Image to 3D Objects with Consistent Multi-View Generation and 3D Diffusion (CVPR 2024 · similarity 0.91)
- DreamCraft3D: Hierarchical 3D Generation with Bootstrapped Diffusion Prior (ICLR 2024 · similarity 0.91)
- Sculpt3D: Multi-View Consistent Text-to-3D Generation with Sparse 3D Prior (CVPR 2024 · similarity 0.89)
- CAD: Photorealistic 3D Generation via Adversarial Distillation (CVPR 2024 · similarity 0.89)
- DIRECT-3D: Learning Direct Text-to-3D Generation on Massive Noisy 3D Data (CVPR 2024 · similarity 0.88)
- PartCrafter: Structured 3D Mesh Generation via Compositional Latent Diffusion Transformers (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.