Research map / 3D vision / Face and shape
CraftsMan3D: High-fidelity Mesh Generation with 3D Native Diffusion and Interactive Geometry Refiner
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Presents CraftsMan3D, a two-stage system that generates coarse meshes with a 3D-native latent set DiT conditioned on multi-view images, then refines surface details via normal-based optimization that can be automatic or user-guided with a brush.
Problem
Single-image and text-to-3D generation struggles with slow optimization, self-occlusion, irregular mesh topology, low surface detail, and lacks support for interactive user editing.
Approach
Combines a 3D-native latent set diffusion transformer conditioned on multi-view images (with camera injection) for coarse geometry with a training-free cross-view normal refinement, Poisson normal blending, and relative Laplacian smoothing to add details, plus a brush-based interactive mask for editing; data is cleaned with a visibility check plus winding-number watertight conversion.
Key results
- On GSO (30 shapes), CraftsMan achieves CD 0.0291 and IoU 0.5347 at about 5s, beating InstantMesh (CD 0.0187 is lower, but IoU 0.6353 higher), so it is not uniformly best; it outperforms all other generative 3D models.
- On a self-occlusion Objaverse subset, CD 0.03943 and IoU 0.53215 vs InstantMesh 0.04909 and 0.50151.
- Mesh refinement reaches CLIP similarity 0.2821 in about 20s vs Fantasia3D 0.2567 (~15min) and Latent-NeRF 0.2725 (~1h).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add multi-view image conditioning instead of single image (ablation) | latent set DiT · GSO subset | CD: 0.0317 -> 0.0188 | Helped |
| add camera pose injection via ModLN (ablation) | multi-view conditioned DiT · GSO subset | CD: 0.0249 -> 0.0188 | Helped |
| use camera injection instead of cost volume (ablation) | multi-view conditioned DiT · GSO subset | CD: 0.0223 -> 0.0188 | Helped |
| add Poisson Normal Blending and relative Laplacian smoothing to normal-based refinement (ablation) | mesh refinement · Qualitative (Figure 10) | visual shape quality | Helped |
| proposed 3D native DiT vs multi-view reconstruction (InstantMesh) on self-occluded subset | image-to-3D generation · Objaverse self-occlusion subset | CD: 0.04909 -> 0.03943 | Helped |
| normal-based refinement vs Fantasia3D and Latent-NeRF | mesh refinement · Objaverse 20 objects | CLIP similarity: 0.2567/0.2725 -> 0.2821 | Helped |
Surprising findings
- On GSO, InstantMesh achieves a lower Chamfer Distance (0.0187) than CraftsMan (0.0291), so the method is not best on every metric.
- Cross-view attention for normal enhancement works training-free without additional training.
Limitations
- Controllability of the latent set diffusion model warrants further investigation.
- Texture generation is not addressed.
Experimental setup
Datasets: Objaverse Google Scanned Objects (GSO) normal dataset [8, 15]
Benchmarks: Google Scanned Objects (GSO) Objaverse subset
Metrics: Chamfer Distance IoU CLIP similarity running time
Baselines: InstantMesh Shap-E Point-E Michelangelo One-2-3-45 zero123 Realfusion Magic123 One2345++ Direct3D Fantasia3D Latent-NeRF
Builds on: Michelangelo CLAY Direct3D ControlNet DiT
Closest papers by meaning
- SILSA: Sliding-Window Slice Latents for Topology-Preserving High-Resolution 3D Generation (NeurIPS 2026 · similarity 0.88)
- ARMesh: Autoregressive Mesh Generation via Next-Level-of-Detail Prediction (NeurIPS 2025 · similarity 0.88)
- GALA: Geometry-Aware Local Adaptive Grids for Detailed 3D Generation (ICLR 2025 · similarity 0.88)
- Rapid 3D Model Generation with Intuitive 3D Input (CVPR 2024 · similarity 0.88)
- Interactive3D: Create What You Want by Interactive 3D Generation (CVPR 2024 · similarity 0.87)
- MeshAnything: Artist-Created Mesh Generation with Autoregressive Transformers (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.