Research map / 3D vision / 3D generation
MIDI: Multi-Instance Diffusion for Single Image to 3D Scene Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
MIDI extends a pre-trained rectified-flow image-to-3D object DiT into a multi-instance diffusion model with multi-instance attention layers, generating all instances of a scene from one image jointly.
Problem
Single-image 3D scene generation requires accurate per-object geometry and spatial layout, but existing methods rely on reconstruction, retrieval, or lengthy multi-stage object-by-object generation that accumulates errors and lacks global scene context.
Approach
Denoise latents of all instances in a scene in parallel with a shared DiT, inserting multi-instance attention layers (K=5 of 21) so each instance token attends to tokens of all instances, conditioned on the global scene image plus per-object images and masks; mixed training with Objaverse single-object data preserves the pre-trained prior.
Key results
- On 3D-Front, MIDI reaches scene-level CD-S 0.080 and F-Score-S 50.19, versus 0.117 / 43.58 for the best prior baseline DiffCAD.
- On BlendSwap, MIDI achieves object-level IoU-B 0.663 versus 0.457 for DiffCAD, with 40s runtime per scene.
- Ablation: K=5 multi-instance attention layers gives the best CD-S (0.080); K=0 gives 0.145 and K=21 gives 0.127; removing global scene image or Objaverse mixing degrades results.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| extend object self-attention with multi-instance attention (K=5) (ablation) | MIDI · 3D-Front | CD-S: 0.145 -> 0.080 | Helped |
| increase multi-instance attention layers from K=5 to K=21 (ablation) | MIDI · 3D-Front | CD-S: 0.080 -> 0.127 | Hurt |
| remove global scene image conditioning (ablation) | MIDI · 3D-Front | CD-S: 0.080 -> 0.134 | Hurt |
| include Objaverse single-object data in mixed training (ablation) | MIDI · 3D-Front | CD-O: 0.126 -> 0.103 | Helped |
| add MIDI designs to scene-data fine-tuning (ablation) | MIDI · 3D-Front | CD-S: 0.152 -> 0.080 | Helped |
Surprising findings
- Using all 21 attention blocks as multi-instance attention (K=21) overfits and distorts object geometry, so a partial subset (K=5) works best.
Limitations
- Performs relatively poorly on tiny-resolution input images.
- Struggles with complex inter-instance interactions, such as characters interacting with objects.
- Evaluated mainly on a limited number of scenes; scaling to many objects and open-world settings is untested.
Experimental setup
Datasets: 3D-Front Objaverse BlendSwap Matterport3D ScanNet
Benchmarks: 3D-Front test set BlendSwap Matterport3D ScanNet SDXL stylized images
Metrics: Chamfer Distance F-Score Volume IoU runtime
Baselines: PanoRecon Total3D InstPIFu SSR DiffCAD Gen3DSR REPARO
Builds on: rectified flow DINO Objaverse
Closest papers by meaning
- Generating compositional scenes via Text-to-image RGBA Instance Generation (NeurIPS 2024 · similarity 0.88)
- LT3SD: Latent Trees for 3D Scene Diffusion (CVPR 2025 · similarity 0.87)
- A 3D Scene is Worth 1K Tokens: 3D-Grounded Representation for Scene Generation at Scale (NeurIPS 2026 · similarity 0.87)
- PartCrafter: Structured 3D Mesh Generation via Compositional Latent Diffusion Transformers (NeurIPS 2025 · similarity 0.87)
- PartGen: Part-level 3D Generation and Reconstruction with Multi-view Diffusion Models (CVPR 2025 · similarity 0.87)
- One-2-3-45++: Fast Single Image to 3D Objects with Consistent Multi-View Generation and 3D Diffusion (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.