Research map / 3D vision / 3D generation
MagicMan: Generative Novel View Synthesis of Humans with 3D-Aware Diffusion and Iterative Refinement
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
MagicMan fine-tunes a Stable Diffusion 1.5 based multi-view diffusion model conditioned on a reference image and SMPL-X guidance to generate 20 dense consistent human novel views and normals from one image, refining SMPL-X iteratively.
Problem
Single-image human novel view synthesis and 3D reconstruction suffer from weak generalization and 3D inconsistency across views, and inaccurate monocular SMPL-X estimates cause ill-shaped geometry.
Approach
Combines a pre-trained 2D diffusion prior with SMPL-X body guidance via a hybrid 1D+sparse-3D multi-view attention for dense views, a dual RGB/normal branch sharing blocks, and an iterative loop in which CFG-scaled generation supervises SMPL-X refinement.
Key results
- On THuman2.1 with 20 views, MagicMan reaches PSNR 24.9 and LPIPS 0.054 versus 23.3 and 0.071 for the next best (Champ*) in the 20-view setting.
- 3D reconstruction Chamfer distance on THuman2.1 is 2.35 versus 4.30 for the best feed-forward baseline (PAMIR).
- Hybrid attention generates up to 20 views in one pass in about 40 s on one A100, while full 3D attention yields only 6 views under the same memory.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MagicMan instead of Champ | human novel view synthesis · THuman2.1 | PSNR: 24.9/23.3 -> 26.0/24.9 | Helped |
| Use MagicMan instead of Champ | human novel view synthesis · CustomHumans | CLIP: 0.938/0.934 -> 0.947/0.947 | Helped |
| Use MagicMan instead of PAMIR | single-image 3D human reconstruction · THuman2.1 | Chamfer: 4.30 -> 2.35 | Helped |
| Add 1D multi-view attention (ablation) | MagicMan baseline · THuman2.1 | PSNR: 22.84 -> 23.94 | Helped |
| Add 3D multi-view attention (ablation) | MagicMan baseline · THuman2.1 | PSNR: 22.84 -> 23.77 | Helped |
| Add hybrid attention instead of 1D attention alone (ablation) | MagicMan · THuman2.1 | LPIPS: 0.063 -> 0.054 | Helped |
| Remove the normal branch (ablation) | MagicMan · THuman2.1 | PSNR: 24.85 -> 23.73 | Hurt |
| Remove the normal branch (ablation) | MagicMan · in-the-wild | CLIP: 0.871 -> 0.833 | Hurt |
| Use reverse 3D-attention view selection instead of the proposed selection (ablation) | MagicMan hybrid attention · THuman2.1 | PSNR: 24.85 -> 23.88 | Hurt |
| Use reverse 3D-attention view selection instead of the proposed selection (ablation) | MagicMan hybrid attention · in-the-wild | CLIP: 0.860 -> 0.871 | Hurt |
| Iteratively refine estimated SMPL-X guidance (ablation) | MagicMan reconstruction · THuman2.1 (50 scans) | Chamfer: 3.73 -> 2.03 | Helped |
| Remove SMPL-X guidance (ablation) | MagicMan reconstruction · THuman2.1 (50 scans) | Chamfer: 3.73 -> 2.96 | Helped |
Surprising findings
- Removing CLIP embeddings from cross-attention was found redundant and discarded, replacing it with reference attention.
- Initial refinement iterations use CFG scale 0 (no SMPL-X guidance) so that generated views match the reference pose better before geometry guidance is increased.
Limitations
- Texture details of reconstructed meshes are not as sharp, since latent-space consistency does not guarantee pixel-level consistency.
- SD1.5 backbone struggles to generate high-quality hands and faces.
- Reconstruction relies on NeuS, which requires strict multi-view consistency.
Experimental setup
Datasets: THuman2.1 CustomHumans SHHQ
Benchmarks: THuman2.1 CustomHumans in-the-wild images
Metrics: PSNR SSIM LPIPS CLIP score Chamfer distance P2S distance normal error
Baselines: Zero123 SyncDreamer Wonder3D SV3D Animate Anyone Champ PIFu PaMIR ICON ECON
Builds on: Stable Diffusion 1.5 Zero-1-to-3 Wonder3D ICON ECON
Closest papers by meaning
- GeneMAN: Generalizable Single-Image 3D Human Reconstruction from Multi-Source Human Data (NeurIPS 2025 · similarity 0.90)
- PSHuman: Photorealistic Single-image 3D Human Reconstruction using Cross-Scale Multiview Diffusion and Explicit Remeshing (CVPR 2025 · similarity 0.89)
- Ouroboros3D: Image-to-3D Generation via 3D-aware Recursive Diffusion (CVPR 2025 · similarity 0.89)
- HumanNorm: Learning Normal Diffusion Model for High-quality and Realistic 3D Human Generation (CVPR 2024 · similarity 0.88)
- DreamComposer: Controllable 3D Object Generation via Multi-View Conditions (CVPR 2024 · similarity 0.88)
- MEAT: Multiview Diffusion Model for Human Generation on Megapixels with Mesh Attention (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.