Research map / 3D vision / 3D generation
HumanCrafter: Synergizing Generalizable Human Reconstruction and Semantic 3D Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes HumanCrafter, a feed-forward framework that jointly reconstructs a textured human from a single image and lifts body-part semantics into the same pixel-aligned 3D Gaussian feature field, using SMPL normal priors, DINOv2 attention-based feature lifting, and a multi-task loss.
Problem
Single-image feed-forward 3D human reconstruction exists but lacks consistent human-part semantic segmentation, and labeled 3D human segmentation data is scarce.
Approach
Combines SMPL-normal geometric priors and SV3D multi-view diffusion features with a Transformer that predicts pixel-aligned 3DGS, and attaches a semantic feature field distilled from frozen DINOv2 via attention-weighted feature propagation, trained jointly with rendering and sparse 2D cross-entropy segmentation losses.
Key results
- On 2K2K with single-view input, HumanCrafter reports PSNR 23.489, SSIM 0.916, LPIPS 0.045, outperforming Human3Diffusion (22.323 PSNR) and other baselines.
- On 2K2K with two ground-truth views, segmentation reaches 0.840 mIoU and 0.925 accuracy, above LSM* (0.724 mIoU) and Sapiens (0.823 mIoU).
- Single-view reconstruction takes about 6.24 s per object versus 23.21 s for Human3Diffusion plus Sapiens; the reconstruction stage alone takes about 0.2 s.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use HumanCrafter instead of Human3Diffusion + Sapiens | single-view human segmentation · 2K2K | mIOU: 0.781 -> 0.801 | Helped |
| use HumanCrafter instead of Human3Diffusion + Sapiens | single-view human segmentation · 2K2K | Acc.: 0.851 -> 0.882 | Helped |
| use HumanCrafter instead of Human3Diffusion | single-image 3D reconstruction · THuman2.1 | PSNR: 22.164 -> 23.186 | Helped |
| use HumanCrafter instead of Human3Diffusion | single-image 3D reconstruction · 2K2K | PSNR: 22.323 -> 23.489 | Helped |
| add SMPL normals (ablation) | HumanCrafter · 2K2K, 1 input view | PSNR: 22.203 -> 23.489 | Helped |
| add the SMPL prior (ablation) | HumanCrafter · 2K2K, 2 input views | PSNR: 23.570 -> 24.786 | Helped |
| use depth instead of SMPL normals (ablation) | HumanCrafter · 2K2K, 1 input view | PSNR: 23.489 -> 23.103 | Hurt |
| use coordinates instead of SMPL normals (ablation) | HumanCrafter · 2K2K, 1 input view | PSNR: 23.324 -> 23.489 | No clear effect |
| add camera embeddings (ablation) | HumanCrafter · 2K2K | PSNR: 23.327 -> 23.489 | Helped |
| add DINOv2 features (ablation) | HumanCrafter · 2K2K | PSNR: 22.025 -> 23.489 | Helped |
| add pixel-aligned aggregation (ablation) | HumanCrafter · 2K2K | PSNR: 21.183 -> 23.489 | Helped |
| add feature distillation loss (ablation) | HumanCrafter · 2K2K | PSNR: 22.464 -> 23.489 | Helped |
Surprising findings
- Fine-tuning the DINOv2 encoder gives only marginal gains (+0.032 PSNR), so keeping it frozen is adequate.
Limitations
- Relies on 2D supervision and a small number of annotated viewpoints (8 per scan for 500 scans) for segmentation.
- Feature rendering limited to up to 1024 dimensions due to GPU memory.
- Single-view reconstruction depends on a 2D diffusion model taking about 6 seconds to generate multi-view latents.
Experimental setup
Datasets: THuman2.1 2K2K Human MVImageNet
Benchmarks: THuman2.1 2K2K
Metrics: mIoU mAcc PSNR SSIM LPIPS COLMAP point count
Baselines: LGM GRM InstantMesh LaRa SiFU PSHuman Human3Diffusion LSM Sapiens
Builds on: 3D Gaussian Splatting LGM LSM SV3D DINOv2
Closest papers by meaning
- GeneMAN: Generalizable Single-Image 3D Human Reconstruction from Multi-Source Human Data (NeurIPS 2025 · similarity 0.87)
- PartCrafter: Structured 3D Mesh Generation via Compositional Latent Diffusion Transformers (NeurIPS 2025 · similarity 0.86)
- MESC-3D:Mining Effective Semantic Cues for 3D Reconstruction from a Single Image (CVPR 2025 · similarity 0.85)
- MultiPly: Reconstruction of Multiple People from Monocular Video in the Wild (CVPR 2024 · similarity 0.85)
- HumanSplat: Generalizable Single-Image Human Gaussian Splatting with Structure Priors (NeurIPS 2024 · similarity 0.85)
- Geometry-Guided Semantic Reconstruction for 3D Instance Segmentation (NeurIPS 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.