Research map / 3D vision / Human pose estimation
UniHOPE: A Unified Approach for Hand-Only and Hand-Object Pose Estimation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes UniHOPE, a single monocular model that estimates hand pose always and object pose only when grasped, using an object switcher, grasp-aware fusion, and diffusion-generated de-occluded pairs with multi-level feature enhancement.
Problem
Monocular 3D hand pose estimation and hand-object pose estimation are handled by separate specialized methods that degrade when applied to the other scenario; a unified method must handle hand-only and hand-object scenes and be robust to object-caused occlusion.
Approach
An internal object switcher predicts grasping status to toggle object output, a grasp-aware feature fusion gates object-to-hand feature interaction by grasp status, and occlusion-invariant features are learned via self-distillation from diffusion-generated de-occluded hand images with adaptive ControlNet strength selection.
Key results
- On DexYCB all-scenes root-relative J-PE, UniHOPE reaches 13.03 mm versus 14.02 for HandOccNet and 14.32 for HFL-Net.
- On FreiHAND cross-dataset root-relative J-PE, UniHOPE achieves 50.97 mm vs 58.02 for HFL-Net and 58.03 for HandOccNet (lower is better; note J-AUC 34.31 vs 29.94).
- Ablation: the full pipeline lowers DexYCB root-relative J-PE from 14.09 (baseline) to 13.03 mm; the object switcher reports >95% grasping status prediction accuracy.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use UniHOPE instead of HandOccNet | hand pose estimation · DexYCB all scenes | J-PE: 14.02 -> 13.03 | Helped |
| Use UniHOPE instead of HFL-Net | hand pose estimation · HO3D | J-PE: 28.45 -> 26.23 | Helped |
| Use UniHOPE instead of HFL-Net | hand pose estimation · FreiHAND test | root-relative J-PE: 58.02 -> 50.97 | Helped |
| Add grasp-aware feature fusion (ablation) | UniHOPE baseline · DexYCB | root-relative J-PE: 14.09 -> 13.84 | Helped |
| Add generative de-occluder and paired de-occluded images (ablation) | UniHOPE with grasp-aware feature fusion · DexYCB | root-relative J-PE: 13.84 -> 13.38 | Helped |
| Add initial-level image feature enhancement (ablation) | UniHOPE with generative de-occluder · DexYCB | root-relative J-PE: 13.38 -> 13.23 | Helped |
| Add RoI-level feature enhancement (ablation) | UniHOPE with image feature enhancement · DexYCB | root-relative J-PE: 13.23 -> 13.18 | Helped |
| Add MANO-level feature enhancement (ablation) | UniHOPE with RoI feature enhancement · DexYCB | root-relative J-PE: 13.18 -> 13.12 | Helped |
| Filter feature-enhancement pairs to grasping and occluded samples (ablation) | UniHOPE with multi-level feature enhancement · DexYCB | root-relative J-PE: 13.12 -> 13.03 | Helped |
Surprising findings
- Training HPE and HOPE methods on all scenes improves hand-object performance but degrades their hand-only performance, showing task-specific designs limit generalization.
Limitations
- Relies on automatically generated grasping labels from object pose change thresholds rather than annotated grasp status.
- Input resolution is 128x128, and feature enhancement needs paired generated images during training.
Experimental setup
Datasets: DexYCB HO3D FreiHAND
Benchmarks: DexYCB HO3D FreiHAND
Metrics: J-PE V-PE PA-J-PE PA-V-PE J-AUC V-AUC F@5 F@15 ADD-0.5D
Baselines: HandOccNet MobRecon H2ONet SimpleHand Liu et al. Keypoint Transformer HFL-Net
Builds on: HFL-Net HandOccNet ControlNet latent diffusion model MANO
Closest papers by meaning
- Harmonious Feature Learning for Interactive Hand-Object Pose Estimation (CVPR 2023 · similarity 0.89)
- EasyHOI: Unleashing the Power of Large Models for Reconstructing Hand-Object Interactions in the Wild (CVPR 2025 · similarity 0.88)
- HOISDF: Constraining 3D Hand-Object Pose Estimation with Global Signed Distance Fields (CVPR 2024 · similarity 0.88)
- ComPose: When to Trust Hands for Object Pose Tracking (NeurIPS 2026 · similarity 0.87)
- StableHand: Quality-Aware Flow Matching for World-Space Dual-Hand Motion Estimation from Egocentric Video (NeurIPS 2026 · similarity 0.86)
- Single-to-Dual-View Adaptation for Egocentric 3D Hand Pose Estimation (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.