Research map / 3D vision / Face and shape
A Simple Baseline for Efficient Hand Mesh Reconstruction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Decomposes hand mesh decoders into a keypoint-guided token generator and a multi-stage upsampling mesh regressor, yielding a simple real-time baseline with SOTA accuracy on FreiHAND and DexYCB.
Problem
Hand mesh reconstruction methods often use complex decoder components that hurt efficiency; the question is which structural elements of the mesh decoder actually matter.
Approach
Decompose the mesh decoder into a token generator (keypoint-guided point sampling at adequate resolution) and a mesh regressor (cascaded metaformer upsampling stages from 21 to 778 tokens with learnable position embeddings), showing that these two core structures suffice for SOTA accuracy with minimal compute.
Key results
- On FreiHAND, Ours (FastViT-MA36) achieves PA-MPJPE 5.7mm and PA-MPVPE 6.0mm at 70 fps; Ours (HRNet) achieves 5.8mm / 6.1mm at 33 fps (abstract cites 5.8/6.1 for both settings).
- On DexYCB, the method reaches PA-MPJPE 5.5mm and PA-MPVPE 5.5mm, improving MPJPE by 1.5mm over prior best (12.4mm vs 14.0mm).
- Ablation: keypoint-guided point sampling at 28x28 plus a 3-layer mesh regressor reduces PA-MPJPE from 6.9mm (simple baseline) to 6.2mm; the proposed approach uses only 1.9M parameters outside the backbone.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace the global-feature token generator and MLP mesh regressor with the proposed mesh regressor (ablation) | Simple Baseline · FreiHAND | PA-MPJPE (mm): 6.9 -> 6.5 | Helped |
| Replace global-feature token generation with the proposed token generator (ablation) | Simple Baseline · FreiHAND | PA-MPJPE (mm): 6.9 -> 6.6 | Helped |
| Use both the proposed token generator and mesh regressor (ablation) | Simple Baseline · FreiHAND | PA-MPJPE (mm): 6.9 -> 6.2 | Helped |
| Use point sampling at 28x28 resolution instead of global features (ablation) | Token generator · FreiHAND | PA-MPJPE (mm): 6.5 -> 6.2 | Helped |
| Use point sampling at 28x28 resolution instead of 7x7 resolution (ablation) | Token generator · FreiHAND | PA-MPJPE (mm): 6.3 -> 6.2 | Helped |
| Replace single four-fold upsampling with two two-fold upsampling layers or add convolutions (ablation) | Token generator · FreiHAND | PA-MPJPE (mm): 6.2 -> 6.2 | No clear effect |
| Use coarse-mesh sampling instead of point sampling (ablation) | Token generator · FreiHAND | PA-MPJPE (mm): 6.2 -> 6.2 | No clear effect |
| Add a second decoder layer with more tokens (ablation) | Mesh regressor · FreiHAND | PA-MPJPE (mm): 6.6 -> 6.3 | Helped |
| Use three decoder layers with progressively increasing token counts (ablation) | Mesh regressor · FreiHAND | PA-MPJPE (mm): 6.6 -> 6.2 | Helped |
| Increase the decoder depth from three to four layers (ablation) | Mesh regressor · FreiHAND | PA-MPJPE (mm): 6.2 -> 6.2 | No clear effect |
| Increase decoder block dimensions from 64, 32, 16 to 256, 128, 64 (ablation) | Mesh regressor · FreiHAND | PA-MPJPE (mm): 6.5 -> 6.2 | Helped |
| Increase the number of blocks per decoder layer from 1, 1, 1 to 3, 3, 3 (ablation) | Mesh regressor · FreiHAND | PA-MPJPE (mm): 6.2 -> 6.3 | No clear effect |
Surprising findings
- Keypoint-guided sampling on the original feature map and coarse-mesh-guided sampling gave no improvement over simpler sampling, and additional convolutions or 4x upsampling schemes performed about the same.
- Increasing the number of blocks or block dimensions in the mesh regressor gave negligible gains; a single block per layer with middle-sized dimensions was optimal.
Limitations
- Designed for single-hand gestures; no improvement reported for extreme lighting, occlusion, hand-object interaction, or out-of-distribution cases.
- Ablations run on smaller Hiera-Tiny backbone rather than the final models.
- Failure cases concentrated in self-occlusion and object occlusion scenes.
Experimental setup
Datasets: FreiHAND DexYCB
Benchmarks: FreiHAND DexYCB
Metrics: PA-MPJPE PA-MPVPE MPJPE MPVPE F-score FPS parameter count
Baselines: I2L-MeshNet CMR I2UV MobRecon METRO MeshGraphormer FastMETRO Deformer PointHMR FastViT HandOccNet H2ONet
Builds on: FastMETRO MobRecon PointHMR MeshGraphormer MetaFormer
Closest papers by meaning
- HandBooster: Boosting 3D Hand-Mesh Reconstruction by Conditional Synthesis and Sampling of Hand-Object Interactions (CVPR 2024 · similarity 0.89)
- Fine-Grained Multi-View Hand Reconstruction Using Inverse Rendering (AAAI 2024 · similarity 0.88)
- Monocular 3D Hand Mesh Recovery via Dual Noise Estimation (AAAI 2024 · similarity 0.87)
- HHMR: Holistic Hand Mesh Recovery by Enhancing the Multimodal Controllability of Graph Diffusion Models (CVPR 2024 · similarity 0.87)
- Reconstructing Hands in 3D with Transformers (CVPR 2024 · similarity 0.87)
- SiMA-Hand: Boosting 3D Hand-Mesh Reconstruction by Single-to-Multi-View Adaptation (AAAI 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.