Research map / 3D vision / 3D generation
IM-3D: Iterative Multiview Diffusion and Reconstruction for High-Quality 3D Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Fine-tunes the Emu Video text-to-video model to generate 16 consistent turntable views of an object, reconstructs them directly with Gaussian splatting using image-level losses, and iterates diffusion and reconstruction twice, avoiding SDS and large reconstruction networks.
Problem
Text-to-3D generation via SDS distillation is slow, unstable, and artifact-prone, while feed-forward reconstruction from multi-view images is limited in quality, requiring large reconstruction networks.
Approach
Replace text-to-image multi-view generators with a fine-tuned video diffusion model (Emu Video) that produces 16 consistent 512px turntable views, fit Gaussian splatting directly to them with LPIPS/SSIM/mask losses, and refine by re-noising renders and regenerating the video a few times.
Key results
- IM-3D achieves CLIP (Image) 92.38 on synthesized views and 91.40 on re-rendered views, the best among compared methods such as MVDream (76.44 / 76.94) and ImageDream (85.39 / 83.77).
- IM-3D requires about 80 network calls per asset versus 320000 for ProlificDreamer and 20000 for MVDream, and takes about 3 minutes per asset.
- Ablation shows replacing LPIPS with RGB loss drops CLIP (Image) from 91.40 to 84.99, and using NeRF instead of Gaussian splatting drops it to 87.37.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use IM-3D instead of MVDream for view synthesis | Text/image-to-3D view synthesis · Synthesized views | CLIP (Text): 31.26 ±2.9 -> 31.92 ±1.6 | Helped |
| Use IM-3D instead of ImageDream for view synthesis | Text/image-to-3D view synthesis · Synthesized views | CLIP (Image): 85.39 ±5.8 -> 92.38 ±5.1 | Helped |
| Use IM-3D instead of MVDream for reconstruction | Text/image-to-3D reconstruction · Re-rendered views | CLIP (Text): 30.63 ±2.7 -> 31.66 ±1.7 | Helped |
| Use IM-3D instead of Zero123XL for reconstruction | Text/image-to-3D reconstruction · Re-rendered views | CLIP (Image): 81.33 ±6.9 -> 91.40 ±5.5 | Helped |
| Add LPIPS loss to the reconstruction objective (ablation) | IM-3D Gaussian Splatting reconstruction · 3D reconstruction | CLIP (Image): 84.71 ±6.4 -> 91.40 ±5.5 | Helped |
| Use LPIPS loss instead of RGB loss (ablation) | IM-3D Gaussian Splatting reconstruction · 3D reconstruction | CLIP (Image): 84.99 ±5.9 -> 91.40 ±5.5 | Helped |
| Add SSIM loss to the reconstruction objective (ablation) | IM-3D Gaussian Splatting reconstruction · 3D reconstruction | CLIP (Image): 90.64 ±5.7 -> 91.40 ±5.5 | Helped |
| Add mask loss to the reconstruction objective (ablation) | IM-3D Gaussian Splatting reconstruction · 3D reconstruction | CLIP (Image): 90.14 ±6.0 -> 91.40 ±5.5 | Helped |
| Use Gaussian Splatting instead of NeRF (ablation) | IM-3D reconstruction · 3D reconstruction | CLIP (Image): 87.37 ±5.4 -> 91.40 ±5.5 | Helped |
| Use 16 frames instead of 8 (ablation) | IM-3D multiview generation and reconstruction · 3D reconstruction with fewer frames | CLIP (Image): 90.06 ±6.3 -> 91.40 ±5.5 | Helped |
| Use 16 frames instead of 4 (ablation) | IM-3D multiview generation and reconstruction · 3D reconstruction with fewer frames | CLIP (Image): 86.96 ±8.6 -> 91.40 ±5.5 | Helped |
Surprising findings
- Extended training of the video model counters its tendency to generate deforming 360 videos, and texture quality did not degrade with longer training unlike MVDream and Instant3D.
Limitations
- Fine-tuned video generator sometimes produces spurious motion (e.g., walking horses) for dynamic subjects, which corrupts reconstruction.
- Relies on a proprietary in-house 3D asset dataset and the closed Emu Video model, limiting reproducibility.
Experimental setup
Datasets: in-house synthetic 3D asset collection (100k subset) prompt set from Magic123/MVDream evaluation
Benchmarks: prompt set from Magic123/MVDream evaluation (39 reconstructions)
Metrics: CLIP text similarity CLIP image similarity human preference win rate number of network calls time per asset
Baselines: MVDream Zero123XL Magic123 SyncDreamer ImageDream OpenLRM One2345++ ProlificDreamer
Builds on: Emu Video Gaussian splatting DPM-Solver++ Score Distillation Sampling
Closest papers by meaning
- IT3D: Improved Text-to-3D Generation with Explicit View Synthesis (AAAI 2024 · similarity 0.91)
- Sharp-It: A Multi-view to Multi-view Diffusion Model for 3D Synthesis and Manipulation (CVPR 2025 · similarity 0.90)
- MVDream: Multi-view Diffusion for 3D Generation (ICLR 2024 · similarity 0.90)
- Cycle3D: High-quality and Consistent Image-to-3D Generation via Generation-Reconstruction Cycle (AAAI 2025 · similarity 0.90)
- DiffSplat: Repurposing Image Diffusion Models for Scalable Gaussian Splat Generation (ICLR 2025 · similarity 0.90)
- ViewDiff: 3D-Consistent Image Generation with Text-to-Image Models (CVPR 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.