Research map / 3D vision / 3D generation
PartGen: Part-level 3D Generation and Reconstruction with Multi-view Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
PartGen decomposes text-, image-, or 3D-derived objects into meaningful parts by fine-tuning a multi-view diffusion model for stochastic part segmentation and another for contextual part completion, followed by a feed-forward reconstruction model.
Problem
Generated or scanned 3D assets are monolithic and unstructured, whereas professional workflows need objects composed of distinct, meaningful, independently editable parts; part segmentation and completion are ambiguous and require hallucinating occluded parts.
Approach
Cast part segmentation as stochastic multi-view color-coded segmentation using a fine-tuned multi-view image generator trained on artist-created assets, and complete each occluded part with a multi-view diffusion model conditioned on the full object context before deterministic 3D reconstruction.
Key results
- PartGen (10 samples) reaches 59.3 mAP50 automatic segmentation vs 35.3 for SAM2 and 11.5 for Part123.
- Contextual part completion achieves CLIP 0.974 and PSNR 21.38 on multi-view parts, beating the no-context and single-view variants.
- Reassembled PartGen objects reach CLIP 0.952 and PSNR 20.33, close to the unstructured whole-object reconstruction (0.955, 20.47).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase the number of segmentation samples from 1 to 10 (ablation) | PartGen · Part segmentation, automatic | mAP50: 45.2 -> 59.3 | Helped |
| Use contextual multi-view completion rather than completion without context (ablation) | PartGen · Part completion, view completion | PSNR: 16.80 -> 21.38 | Helped |
| Use joint multi-view completion rather than completing each view independently (ablation) | PartGen · Part completion, view completion | PSNR: 15.92 -> 21.38 | Helped |
| Add part completion instead of using the masked input views directly (ablation) | PartGen · Part completion, view completion | PSNR: 13.24 -> 21.38 | Helped |
| Decompose and reassemble the object into parts rather than reconstructing it as one unstructured object | PartGen · Reassembling parts | CLIP: 0.955 vs 0.952 | No clear effect |
Surprising findings
- Fine-tuned SAM2 on the same data underperforms the zero-shot-style multi-view diffusion approach by a large margin in segmentation, suggesting generative segmentation better captures artist intent.
Limitations
- Segmentation mAP remains low overall due to inherent ambiguity of part decomposition.
- Relies on a proprietary licensed artist-created dataset that is not released.
- Reconstruction model is deterministic and not fine-tuned for parts, so reconstruction quality is bounded by the RM.
Experimental setup
Datasets: 140k licensed artist-created 3D assets (filtered to 45k objects, 210k parts) Google Scanned Objects
Benchmarks: held-out 100 artist-created objects (internal) Google Scanned Objects (qualitative)
Metrics: mAP50 mAP75 PSNR LPIPS CLIP similarity
Baselines: SAM2 Part123 SAM2 fine-tuned on data SAM2 multi-view fine-tuned
Builds on: AssetGen Instant3D Emu LightplaneLRM SAM2
Closest papers by meaning
- PartCrafter: Structured 3D Mesh Generation via Compositional Latent Diffusion Transformers (NeurIPS 2025 · similarity 0.93)
- AutoPartGen: Autoregressive 3D Part Generation and Discovery (NeurIPS 2025 · similarity 0.92)
- Efficient Part-level 3D Object Generation via Dual Volume Packing (NeurIPS 2025 · similarity 0.90)
- PartNeXt: A Next-Generation Dataset for Fine-Grained and Hierarchical 3D Part Understanding (NeurIPS 2025 · similarity 0.89)
- Learning Partonomic 3D Reconstruction from Image Collections (CVPR 2025 · similarity 0.89)
- LT3SD: Latent Trees for 3D Scene Diffusion (CVPR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.