Research map / 3D vision / 3D generation
PartCrafter: Structured 3D Mesh Generation via Compositional Latent Diffusion Transformers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces PartCrafter, a compositional latent diffusion transformer that jointly generates multiple part meshes from one RGB image without segmentation, initialized from TripoSG and trained on a curated part-annotated dataset.
Problem
Single-image 3D generation typically produces monolithic meshes, and existing part-aware pipelines rely on a two-stage segment-then-reconstruct process that is error-prone, costly, and cannot recover occluded parts.
Approach
Expands a pretrained 3D object latent diffusion transformer (TripoSG) into a compositional model with a separate latent token set per part, a learnable part identity embedding, and alternating local (intra-part) and global (inter-part) attention, with DINOv2 image cross-attention in both, so parts are denoised jointly and in one shot.
Key results
- On Objaverse part-level generation, PartCrafter reaches CD 0.1726 and F-Score 0.7472 versus HoloPart's 0.1916 and 0.6916, with generation time of 34s versus 18min.
- On 3D-Front scenes, PartCrafter achieves CD 0.1491 and F-Score 0.8148 versus MIDI's 0.1602 and 0.7931, without using ground-truth segmentation masks that MIDI uses.
- On the occluded 3D-Front subset, PartCrafter's CD is 0.1508 versus MIDI's 0.2591, showing better handling of invisible parts.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use PartCrafter instead of HoloPart | 3D part-level object generation · Objaverse, ShapeNet, and ABO | CD: Objaverse: 0.1916 -> 0.1726; ShapeNet: 0.3511 -> 0.3205; ABO: 0.1338 -> 0.1047 | Helped |
| Use PartCrafter instead of MIDI | 3D scene generation · 3D-Front | CD: 0.1602 -> 0.1491 | Helped |
| Use PartCrafter instead of MIDI | 3D scene generation · 3D-Front (Occluded) | CD: 0.2591 -> 0.1508 | Helped |
| Use PartCrafter instead of MIDI | 3D scene generation · 3D-Front | IoU: 0.0013 -> 0.0034 | Hurt |
| Remove part identity embeddings (ablation) | PartCrafter · Objaverse | F-Score: 0.7374 -> 0.4978 | Hurt |
| Remove local self-attention (ablation) | PartCrafter · Objaverse | F-Score: 0.7374 -> 0.2327 | Hurt |
| Remove global self-attention (ablation) | PartCrafter · Objaverse | IoU: 0.0814 -> 0.1602 | Hurt |
| Remove cross-attention from local attention (ablation) | PartCrafter · Objaverse | CD: 0.1711 -> 0.1847 | Hurt |
| Remove cross-attention from global attention (ablation) | PartCrafter · Objaverse | IoU: 0.0814 -> 0.0869 | Hurt |
| Place global attention at the sides instead of in the middle (ablation) | PartCrafter · Objaverse | IoU: 0.0814 -> 0.0336 | Helped |
| Place global attention in an alternating pattern instead of in the middle (ablation) | PartCrafter · Objaverse | F-Score: 0.7374 -> 0.7212 | Hurt |
Surprising findings
- Removing part identity embeddings causes the model to collapse, and removing local attention yields no meaningful mesh.
- PartCrafter outperforms its own backbone TripoSG on object-level metrics even with token count aligned.
Limitations
- Trained on only about 50K part-level objects, much smaller than typical object-generation training sets.
- Performs worse on ShapeNet, attributed to the backbone's degraded performance there.
- MIDI slightly surpasses PartCrafter on IoU, since MIDI uses ground-truth segmentation masks.
Experimental setup
Datasets: Objaverse ShapeNet Amazon Berkeley Objects 3D-Front LGM Objaverse subset
Benchmarks: Objaverse ShapeNet ABO 3D-Front 3D-Front (Occluded)
Metrics: Chamfer Distance F-Score IoU generation time
Baselines: HoloPart TripoSG MIDI
Builds on: TripoSG 3DShape2VecSet rectified flow DINOv2 HoloPart
Closest papers by meaning
- PartGen: Part-level 3D Generation and Reconstruction with Multi-view Diffusion Models (CVPR 2025 · similarity 0.93)
- AutoPartGen: Autoregressive 3D Part Generation and Discovery (NeurIPS 2025 · similarity 0.91)
- LT3SD: Latent Trees for 3D Scene Diffusion (CVPR 2025 · similarity 0.90)
- Learning Partonomic 3D Reconstruction from Image Collections (CVPR 2025 · similarity 0.90)
- MeshArt: Generating Articulated Meshes with Structure-Guided Transformers (CVPR 2025 · similarity 0.89)
- Structured 3D Latents for Scalable and Versatile 3D Generation (CVPR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.