Research map / 3D vision / 3D generation
SiMA-Hand: Boosting 3D Hand-Mesh Reconstruction by Single-to-Multi-View Adaptation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SiMA-Hand, which trains a single-view 3D hand-mesh reconstructor with distilled shape and orientation features from a multi-view reconstructor, so only one view is needed at inference.
Problem
3D hand-mesh reconstruction from monocular RGB images degrades severely under heavy self- or object-caused occlusion, while multi-view or multi-frame methods need extra views or frames at inference.
Approach
Train a multi-view hand reconstructor (MVR-Hand) with image-, joint-, and vertex-level fusion, then distill its shape (vertex-level) and orientation features into a single-view reconstructor (SVR-Hand) via shape and orientation feature enhancement losses, so inference uses one view only.
Key results
- On Dex-YCB, SiMA-Hand achieves PA-VPE 4.98 mm and PA-JPE 5.16 mm, best among compared methods including H2ONet (5.45 / 5.65).
- On HanCo (after PA), SiMA-Hand reaches JPE 6.55 mm and VPE 6.48 mm, improving over H2ONet (6.78 / 6.89).
- On occluded Dex-YCB frames, SiMA-Hand obtains PA-VPE 6.18 mm and PA-F@5 73.09 versus H2ONet's 6.42 and 71.73.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use SiMA-Hand instead of H2ONet | Dex-YCB single-view hand reconstruction · Dex-YCB | PA-JPE: 5.65 -> 5.16 | Helped |
| use SiMA-Hand instead of H2ONet | Dex-YCB single-view hand reconstruction · Dex-YCB | VPE: 13.03 -> 12.78 | Helped |
| use SiMA-Hand instead of H2ONet | HanCo hand reconstruction · HanCo (after PA) | JPE: 6.78 -> 6.55 | Helped |
| use SiMA-Hand instead of H2ONet | Dex-YCB occluded hand reconstruction · Dex-YCB, occluded | VPE: 18.47 -> 18.29 | Helped |
| add image-level feature fusion (IFF) (ablation) | MVR-Hand · Dex-YCB | PA-JPE: 3.56 -> 3.57 | Helped |
| add joint-level feature fusion (JFF) (ablation) | MVR-Hand · Dex-YCB | JPE: 9.48 -> 8.49 | Helped |
| use JFF pooling-based fusion instead of concatenation (ablation) | MVR-Hand · Dex-YCB | JPE: 8.55 -> 8.49 | Helped |
| use attention-based VFF instead of concatenation (ablation) | MVR-Hand · Dex-YCB | PA-VPE: 4.21 -> 3.87 | Helped |
| use attention-based VFF instead of pooling (ablation) | MVR-Hand · Dex-YCB | PA-VPE: 3.77 -> 3.87 | Hurt |
| add orientation feature enhancement (OFE) (ablation) | SVR-Hand · Dex-YCB | JPE: 13.66 -> 13.37 | Helped |
| use OFE attention instead of concatenation (ablation) | SVR-Hand · Dex-YCB | JPE: 13.59 -> 13.37 | Helped |
| add feature-enhancement constraints through SiMA (ablation) | SVR-Hand · Dex-YCB | VPE: 13.11 -> 12.78 | Helped |
Surprising findings
- Pooling-based vertex fusion gives better PA-VPE for the multi-view reconstructor but hurts under the SiMA framework due to information loss.
- Distilling the shared joint-level feature hurts orientation estimation, while distilling the enhanced orientation feature gives the largest gain.
Limitations
- Single-view model has 78.42M parameters and 1.65 GFLOPs, larger than MobRecon and H2ONet, and runs slower (48 FPS).
- Training requires synchronized multi-view data (8 views), which is not needed at inference but is needed for training.
- Evaluated only on Dex-YCB and HanCo; occlusion splits follow prior work.
Experimental setup
Datasets: Dex-YCB HanCo FreiHAND
Benchmarks: Dex-YCB HanCo
Metrics: JPE VPE JAUC VAUC F@5 F@15 PA-JPE PA-VPE PA-JAUC PA-VAUC
Baselines: Spurr et al. METRO Liu et al. Tse et al. HandOccNet MobRecon H2ONet I2L-MeshNet POEM
Builds on: HandOccNet H2ONet POEM MobRecon FreiHAND
Closest papers by meaning
- Fine-Grained Multi-View Hand Reconstruction Using Inverse Rendering (AAAI 2024 · similarity 0.90)
- HandBooster: Boosting 3D Hand-Mesh Reconstruction by Conditional Synthesis and Sampling of Hand-Object Interactions (CVPR 2024 · similarity 0.89)
- SRHand: Super-Resolving Hand Images and 3D Shapes via View/Pose-aware Neural Image Representations and Explicit Meshes (NeurIPS 2025 · similarity 0.89)
- HandOS: 3D Hand Reconstruction in One Stage (CVPR 2025 · similarity 0.88)
- Learning Explicit Contact for Implicit Reconstruction of Hand-Held Objects from Monocular Images (AAAI 2024 · similarity 0.87)
- HOLD: Category-agnostic 3D Reconstruction of Interacting Hands and Objects from Video (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.