Research map / 3D vision / Face and shape
Semantic Human Mesh Reconstruction with Textures
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SHERT, a pipeline that converts detailed human surfaces (mesh/SDF) into fully textured semantic SMPL-X-based meshes via semantic/normal-based sampling, self-supervised UV-domain completion and refinement networks, and a ControlNet-finetuned Stable Diffusion texture model.
Problem
Monocular and scan-based 3D human mesh reconstruction produces unstable, low-quality meshes lacking consistent UV unwrapping, skinning weights and semantic labels, which limits industrial use such as animation and texture editing.
Approach
Use SMPL-X (subdivided to ~150k vertices) as a semantic guider: sample the detailed target surface along vertex normals, then complete and refine the partial result in a canonical UV position-map domain treated as 2D inpainting, and generate/inpaint UV textures with a text- and image-conditioned diffusion model so every vertex keeps fixed semantics and skinning weights.
Key results
- On THuman2.0 monocular reconstruction, Ours (ICON-refine) reaches P2S 1.0442 and Chamfer 1.0468 versus ICON's 1.0361 and 1.0874, so gains are mixed rather than uniform.
- On registration (THuman2.0 first subject), Ours (SNS) gives P2S 0.107 and Chamfer 0.078, better than Fast-RNRR (0.115/0.097), with G-avg 0.729 and runtime 23s versus 1m 4s for Fast-RNRR.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Complete ICON-predicted meshes | ICON predictions · CAPE | P2S: 0.8855 -> 0.8550 | Helped |
| Complete ICON-predicted meshes | ICON predictions · THuman2.0 | Chamfer: 1.0874 -> 1.0465 | Helped |
| Refine ECON-based meshes | ECON predictions · THuman2.0 | Chamfer: 1.2081 -> 1.0430 | Helped |
| Refine ECON-based meshes | ECON predictions · THuman2.0 | Normal: 0.0661 -> 0.0649 | Helped |
| Iteratively apply the refinement network (ablation) | SHERT completed mesh · Mesh refinement | mesh detail | Helped |
| Use the completion network after SNS (ablation) | SNS registration on THuman2.0 · THuman2.0 | P2S: 0.107 -> 0.139 | Hurt |
Surprising findings
- Some ECON/ICON-based variants of SHERT are slightly worse than their base methods on THuman2.0 (e.g., ECON-comp P2S 1.1255 vs ECON 1.1304 is better, but Chamfer for ECON-comp 1.1420 is worse than ECON 1.2081 only in some cases), indicating inconsistent gains across datasets.
Limitations
- Weaker than implicit methods for loose clothing, shoes and hair due to SMPL-X geometric limits
- Texture diffusion has difficulty keeping consistency at UV seams
Experimental setup
Datasets: THuman2.0 CAPE
Benchmarks: CAPE-NFP THuman2.0
Metrics: P2S Chamfer distance normal difference G-avg theta<30 degrees percentage runtime
Baselines: PIFu PIFuHD PaMIR ICON ECON 2K2K N-ICP RPTS SVR-L0 Fast-RNRR DR MDA
Builds on: SMPL-X ECON ICON ControlNet Stable Diffusion DINAR
Closest papers by meaning
- SCULPT: Shape-Conditioned Unpaired Learning of Pose-dependent Clothed and Textured Human Meshes (CVPR 2024 · similarity 0.86)
- HISR: Hybrid Implicit Surface Representation for Photorealistic 3D Human Reconstruction (AAAI 2024 · similarity 0.85)
- TriTex: Learning Texture from a Single Mesh via Triplane Semantic Features (CVPR 2025 · similarity 0.85)
- PSHuman: Photorealistic Single-image 3D Human Reconstruction using Cross-Scale Multiview Diffusion and Explicit Remeshing (CVPR 2025 · similarity 0.85)
- MESC-3D:Mining Effective Semantic Cues for 3D Reconstruction from a Single Image (CVPR 2025 · similarity 0.85)
- StdGEN: Semantic-Decomposed 3D Character Generation from Single Images (CVPR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.