Research map / Image generation / Fast diffusion sampling
HHMR: Holistic Hand Mesh Recovery by Enhancing the Multimodal Controllability of Graph Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes HHMR, an all-in-one graph diffusion model (GCN + self/cross-attention U-Net) that performs hand mesh generation, inpainting, reconstruction and 2D-skeleton fitting from multimodal conditions, with modality/feature-level random masking and condition-aligned gradient guidance.
Problem
Hand mesh recovery is usually handled by one-way regression or by implicit hand priors that need inversion or fitting; a single framework that supports generation, inpainting, reconstruction and fitting is lacking.
Approach
A graph-convolution and attention diffusion U-Net that maps image, 2D skeleton and 3D skeleton conditions into a shared feature space, trained with modality- and feature-level random masking, and steered at inference by a gradient guidance term that fits the generated mesh to the observed condition without an extra classifier.
Key results
- On FreiHAND single-hypothesis reconstruction, MPJPE 6.0 / MPVPE 6.0 (n=1), comparable to MeshGraphormer (5.9/6.0).
- Multi-hypothesis with n=8 gives MPJPE 5.9 / MPVPE 5.9, which outperforms MobRecon (6.1/6.2) and MeshGraphormer (5.9/6.0) at n=1 baseline comparisons; n=16 reaches 5.8.
- Hand generation APD 17.30 vs 15.00 for PCA sampling, with SI 0.05% vs 0.39%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace GCN layers with self-attention layers (ablation) | HHMR · FreiHand | MPJPE: 5.99 -> 6.39 | Hurt |
| replace GCN layers with self-attention layers (ablation) | HHMR · FreiHand | MPVPE: 5.97 -> 6.35 | Hurt |
| add condition-aligned gradient guidance to 2D skeleton fitting | HHMR with 2D skeleton condition · Hand mesh fitting | MPJPE: 7.27 -> 6.26 | Helped |
| add condition-aligned gradient guidance to image-and-skeleton conditioning | HHMR with image and 2D skeleton conditions · Hand mesh fitting | MPJPE: 5.29 -> 4.94 | Helped |
| add 2D skeleton condition to image conditioning | HHMR · Hand mesh fitting | MPJPE: 6.57 -> 5.29 | Helped |
| increase the number of reconstruction samples from 8 to 32 at T=10 (ablation) | HHMR · FreiHand | MPJPE: 5.91 -> 5.63 | Helped |
| increase the number of reconstruction samples from 8 to 32 at T=10 (ablation) | HHMR · FreiHand | MPVPE: 5.90 -> 5.64 | Helped |
| use graph diffusion generation instead of PCA sampling | HHMR · Hand mesh generation | APD (mm): 15.00 -> 17.30 | Helped |
| use graph diffusion generation instead of PCA sampling | HHMR · Hand mesh generation | SI (%): 0.39 -> 0.05 | Helped |
| use graph diffusion inpainting instead of PCA inpainting | HHMR · Hand mesh inpainting | APD (mm): 12.28 -> 13.59 | Helped |
| use graph diffusion inpainting instead of PCA inpainting | HHMR · Hand mesh inpainting | SI (%): 0.06 -> 0.01 | Helped |
Surprising findings
- Replacing GCN layers with self-attention only increases MPJPE from 5.99 to 6.39 on FreiHAND, showing the GCN component matters.
Limitations
- May fail under extremely missing or excessively noisy conditions.
- Increasing denoising steps for more precise generation is time-consuming.
Experimental setup
Datasets: Two-Hand 500K InterHand2.6M FreiHAND HO3D V3 CompHand
Benchmarks: FreiHAND Tab.1 PCA generation/inpainting comparison (APD, SI)
Metrics: MPJPE MPVPE APD SI (self-intersection %)
Baselines: MobRecon MeshGraphormer PCA sampling (MANO) METRO I2UV-HandNet CMR Tang et al.
Builds on: DDIM classifier guidance MeshGraphormer MANO ResNet50
Closest papers by meaning
- Fine-Grained Multi-View Hand Reconstruction Using Inverse Rendering (AAAI 2024 · similarity 0.89)
- HandBooster: Boosting 3D Hand-Mesh Reconstruction by Conditional Synthesis and Sampling of Hand-Object Interactions (CVPR 2024 · similarity 0.89)
- RHanDS: Refining Malformed Hands for Generated Images with Decoupled Structure and Style Guidance (AAAI 2025 · similarity 0.87)
- A Simple Baseline for Efficient Hand Mesh Reconstruction (CVPR 2024 · similarity 0.87)
- Reconstructing Hands in 3D with Transformers (CVPR 2024 · similarity 0.87)
- SiMA-Hand: Boosting 3D Hand-Mesh Reconstruction by Single-to-Multi-View Adaptation (AAAI 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.