Research map / 3D vision / Face and shape
Fine-Grained Multi-View Hand Reconstruction Using Inverse Rendering
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Multi-view hand reconstruction pipeline that initializes MANO with a GCN, refines a subdivided mesh and albedo via inverse rendering (HAM module), and jointly optimizes mesh and a mesh-based neural renderer for photo-realistic rendering.
Problem
Accurately recovering high-fidelity hand geometry and intricate textures from sparse multi-view images, where parametric models are over-smoothed and learning-based methods over-smooth and need lots of data.
Approach
Combines a GCN-regressed MANO initialization with a Hand Albedo and Mesh (HAM) inverse-rendering refinement on a subdivided mesh (Lambertian + spherical harmonics lighting, Laplacian and edge regularizers), then fuses a pre-trained neural renderer with learnable vertex features and jointly optimizes geometry with a geometry-based shader.
Key results
- On InterHand2.6M test/Capture0-Single, the method reports PSNR 37.26 and LPIPS 0.0069, versus 35.34 and 0.0197 for NeuS.
- On DeepHandMesh (5 views), mean P2S drops to 1.456 mm from 2.937 for DHM and 1.873 without HAM.
- Full pipeline takes about 90 seconds on a single NVIDIA 3090Ti, with fine-tuning around one minute.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the full method instead of NeuS | Fine-Grained Multi-View Hand Reconstruction · InterHand2.6M test/Capture0-Single | PSNR: 35.34 -> 37.26 | Helped |
| Use the full method instead of NeuS | Fine-Grained Multi-View Hand Reconstruction · InterHand2.6M test/Capture1-Single | PSNR: 34.98 -> 37.91 | Helped |
| Use the full method instead of NeuS | Fine-Grained Multi-View Hand Reconstruction · InterHand2.6M test/Capture0-Interacting | PSNR: 31.46 -> 37.08 | Helped |
| Add HAM to the initialized mesh (ablation) | Fine-Grained Multi-View Hand Reconstruction · Evaluation (P2S and PSNR) | P2S: 1.873 -> 1.457 | Helped |
| Add HAM to the initialized mesh (ablation) | Fine-Grained Multi-View Hand Reconstruction · DeepHandMesh (5 views), Total Average | P2S: 1.873 -> 1.456 | Helped |
| Add coarse initialization (ablation) | Fine-Grained Multi-View Hand Reconstruction · Evaluation (P2S and PSNR) | P2S: 5.506 -> 1.456 | Helped |
| Add joint refinement (ablation) | Fine-Grained Multi-View Hand Reconstruction · Evaluation (P2S and PSNR) | PSNR: 35.68 -> 37.91 | Helped |
| Add joint refinement (ablation) | Fine-Grained Multi-View Hand Reconstruction · Evaluation (P2S and PSNR) | P2S: 1.457 -> 1.456 | Helped |
| Remove coarse initialization (ablation) | HAM + joint refinement · Evaluation (P2S and PSNR) | PSNR: 37.91 -> 31.96 | Hurt |
Surprising findings
- Adding HAM and joint refinement without the GCN initialization yields a P2S of 5.506 mm, far worse than the 1.873 mm of the initialization alone.
Limitations
- Mesh-based rendering gives semi-transparent mist-like artifacts compared with NeuS in some views.
- Results on DeepHandMesh rely on five views, and NeuS fails with fewer views.
Experimental setup
Datasets: InterHand2.6M DeepHandMesh
Benchmarks: InterHand2.6M DeepHandMesh
Metrics: PSNR SSIM LPIPS P2S
Baselines: Ani-NeRF HandAvatar HandNeRF NeuS DeepHandMesh
Builds on: MANO NeuS HandAvatar DeepHandMesh HandNeRF
Closest papers by meaning
- SiMA-Hand: Boosting 3D Hand-Mesh Reconstruction by Single-to-Multi-View Adaptation (AAAI 2024 · similarity 0.90)
- HandBooster: Boosting 3D Hand-Mesh Reconstruction by Conditional Synthesis and Sampling of Hand-Object Interactions (CVPR 2024 · similarity 0.90)
- SRHand: Super-Resolving Hand Images and 3D Shapes via View/Pose-aware Neural Image Representations and Explicit Meshes (NeurIPS 2025 · similarity 0.90)
- Reconstructing Hands in 3D with Transformers (CVPR 2024 · similarity 0.89)
- HHMR: Holistic Hand Mesh Recovery by Enhancing the Multimodal Controllability of Graph Diffusion Models (CVPR 2024 · similarity 0.89)
- Learning Explicit Contact for Implicit Reconstruction of Hand-Held Objects from Monocular Images (AAAI 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.