Research map / 3D vision / Face and shape
SyncHuman: Synchronizing 2D and 3D Generative Models for Single-view Human Reconstruction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Jointly fine-tunes a 2D multiview generative model (PSHuman/SD2.1) and the native 3D generator Trellis with 2D-3D synchronization attention, then uses a multiview guided decoder to lift 2D detail onto 3D human meshes from a single image.
Problem
Single-view 3D clothed human reconstruction struggles with challenging poses because SMPL-conditioned multiview generators get inaccurate 3D priors, while native 3D generators lack fine detail and input fidelity.
Approach
Couple a multiview (2D) generator and a native 3D sparse-structure generator through pixel-aligned 2D-3D synchronization attention so they co-generate geometrically aligned shapes and views, then inject multiview DINOv2 features into the structured-latent decoder to recover fine details.
Key results
- On CAPE-NFP, the method reaches Chamfer 0.9127 and P2S 0.8113 vs. PSHuman's 1.3726 and 0.9863 (lower is better).
- On X-Humans, PSNR 21.84 and LPIPS 0.0786 vs. PSHuman's 20.84 and 0.098, and TRELLIS's 17.08 and 0.153.
- Ablation: 3D structure IoU rises from 0.4813 without 2D-3D synchronization attention to 0.5907 with it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Integrate synchronized 2D multiview and 3D generation (ablation) | SyncHuman · X-Humans | Chamfer Distance: 1.438 -> 0.835 | Helped |
| Fine-tune Trellis on human scans (ablation) | Trellis · X-Humans | Chamfer Distance: 2.004 -> 1.135 | Helped |
| Add multiview-guided feature injection to the decoder (ablation) | Trellis decoder · X-Humans | PSNR: 21.362 -> 21.838 | Helped |
| Add 2D-3D synchronization attention (ablation) | SyncHuman intermediate 3D structure generation · X-Humans intermediate 3D structure | IoU: 0.4813 -> 0.5907 | Helped |
| Add 2D-3D attention (ablation) | SyncHuman intermediate image generation · X-Humans intermediate color images | PSNR: 23.328 -> 24.027 | Helped |
| Add 2D-3D attention (ablation) | SyncHuman intermediate image generation · X-Humans intermediate normal images | PSNR: 22.851 -> 23.439 | Helped |
| Use SyncHuman instead of PSHuman | Single-view human reconstruction · CAPE-NFP | Chamfer Distance: 1.3726 -> 0.9127 | Helped |
| Use SyncHuman instead of PSHuman | Single-view human reconstruction · CAPE-FP | Chamfer Distance: 0.7764 -> 0.6409 | Helped |
| Use SyncHuman instead of PSHuman | Single-view human reconstruction · X-Humans | PSNR: 20.8405 -> 21.8385 | Helped |
Surprising findings
- Inaccurate SMPL estimates (e.g., self-intersections from 4D-Humans) are a bottleneck, and a native 3D generator's structure aligns better with inputs for challenging poses.
Limitations
- Textures show artifacts under non-uniform or extreme lighting because training renders use uniform lighting.
- Multiview model is fine-tuned from SD 2.1 on only about 5,000 human scans, which constrains generation quality.
- Output meshes may contain holes because the FlexiCubes-based mesh decoder does not enforce watertightness.
Experimental setup
Datasets: THuman2.1 CustomHumans THuman3.0 2K2K X-Humans CAPE
Benchmarks: X-Humans CAPE-FP CAPE-NFP
Metrics: Chamfer distance P2S normal consistency PSNR SSIM LPIPS IoU
Baselines: ICON ECON GTA SIFU SiTH Human3Diff PSHuman TRELLIS
Builds on: PSHuman TRELLIS Stable Diffusion 2.1 flow matching DINOv2
Closest papers by meaning
- SiTH: Single-view Textured Human Reconstruction with Image-Conditioned Diffusion (CVPR 2024 · similarity 0.91)
- PSHuman: Photorealistic Single-image 3D Human Reconstruction using Cross-Scale Multiview Diffusion and Explicit Remeshing (CVPR 2025 · similarity 0.91)
- DiffHuman: Probabilistic Photorealistic 3D Reconstruction of Humans (CVPR 2024 · similarity 0.89)
- Joint2Human: High-Quality 3D Human Generation via Compact Spherical Embedding of 3D Joints (CVPR 2024 · similarity 0.89)
- GeneMAN: Generalizable Single-Image 3D Human Reconstruction from Multi-Source Human Data (NeurIPS 2025 · similarity 0.88)
- HyperHuman: Hyper-Realistic Human Generation with Latent Structural Diffusion (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.