Research map / 3D vision / Human pose estimation
HOISDF: Constraining 3D Hand-Object Pose Estimation with Global Signed Distance Fields
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes HOISDF, a monocular hand-object pose estimator that jointly learns hand and object signed distance fields as implicit global intermediate representations and uses them to sample, augment, and regress hand and object poses, reaching SOTA on DexYCB and HO3Dv2.
Problem
Joint 3D hand-object pose estimation from a single monocular image is hard under frequent occlusion; existing intermediate shape representations (keypoints, meshes, point clouds) are explicit and only provide information near the current pose estimate.
Approach
Learn hand and object signed distance fields in original camera space as an implicit global intermediate representation, then use the fields to sample informative query points, augment point features via volume density, add cross-target hand-object SDF cues, and regress poses with point-wise attention.
Key results
- On DexYCB (DexYCB split), HOISDF reaches MJE 10.1 mm, OCE 18.4 mm and ADD-S 13.3 mm, versus 11.9, 39.8 and 31.9 mm for the prior best Lin et al. on object metrics (Table 2 values for Lin et al. object ADD-S 31.9).
- On HO3Dv2, HOISDF* (with synthetic data) reaches MJE 19.0 mm and OME 35.5 mm, beating Wang et al.* (22.2 mm MJE, 45.5 mm OME).
- Replacing SDF intermediate representation with 3D vertices reduces accuracy (MJE 12.7 mm vs 10.1 mm for SDFs) on DexYCB ablation.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use global 3D SDFs as the intermediate representation instead of 2D keypoints (ablation) | HOISDF · DexYCB test | MJE: 14.9 -> 10.1 | Helped |
| use global 3D SDFs as the intermediate representation instead of 2D segmentation (ablation) | HOISDF · DexYCB test | MJE: 14.1 -> 10.1 | Helped |
| use global 3D SDFs as the intermediate representation instead of 3D vertices (ablation) | HOISDF · DexYCB test | MJE: 12.7 -> 10.1 | Helped |
| use HOISDF instead of Wang et al. | hand-object pose estimation · DexYCB test | MJE: 12.7 -> 10.1 | Helped |
| use HOISDF instead of Lin et al. | hand-object pose estimation · HO3Dv2 test | PAMJE: 8.9 -> 9.6 | Hurt |
| use HOISDF instead of the best listed baseline | hand-object pose estimation · DexYCB Full test | MJE: 12.6 -> 10.1 | Helped |
| use HOISDF instead of the best listed baseline | hand-object pose estimation · HO3Dv2 test | MJE: 25.5 -> 23.6 | Helped |
| use HOISDF instead of the best listed baseline | hand-object pose estimation with synthetic co-training · HO3Dv2 test | MJE: 22.2 -> 19.0 | Helped |
| sample query points using absolute distance rather than random sampling (ablation) | HOISDF · DexYCB test | MJE: 25.8 -> 10.1 | Helped |
| sample query points using absolute distance rather than signed distance (ablation) | HOISDF · DexYCB test | MJE: 13.3 -> 10.1 | Helped |
| remove SDF-based point feature augmentation (ablation) | HOISDF · DexYCB test | MJE: 10.1 -> 11.5 | Hurt |
| replace SDF augmentation with density concatenation (ablation) | HOISDF · DexYCB test | MJE: 10.1 -> 11.0 | Hurt |
Surprising findings
- Hand pose regression remains accurate with simple targets: removing intermediate joint regression or MANO regression only mildly degrades results (MJE 10.4 and 10.5 mm vs 10.1 mm).
Limitations
- Training relies on ground-truth mesh access to sample query points near surfaces; inference uses predicted fields.
- Evaluated only on two benchmarks (DexYCB, HO3Dv2) with a single-view setting.
- HO3Dv2 results with synthetic data require an external rendering pipeline.
Experimental setup
Datasets: DexYCB HO3Dv2
Benchmarks: DexYCB HO3Dv2
Metrics: MJE PAMJE OCE MCE ADD-S MME VAUC F@5mm F@15mm STMJE
Baselines: Lin et al. (harmonious feature learning) Wang et al. (dense mutual attention) Chen et al. (SDF-based hand-object reconstruction, ArtiBoost-style) Chen et al. (kinematic chain SDF) Hampali et al. Tze et al. Hasson et al. Li et al.
Builds on: StyleSDF Hampali et al. transformer hand-object pose (HOT-style cross-attention) Wang et al. dense mutual attention DeepSDF-style neural distance fields
Closest papers by meaning
- UniHOPE: A Unified Approach for Hand-Only and Hand-Object Pose Estimation (CVPR 2025 · similarity 0.88)
- HOLD: Category-agnostic 3D Reconstruction of Interacting Hands and Objects from Video (CVPR 2024 · similarity 0.88)
- HandDiff: 3D Hand Pose Estimation with Diffusion on Image-Point Cloud (CVPR 2024 · similarity 0.87)
- HandBooster: Boosting 3D Hand-Mesh Reconstruction by Conditional Synthesis and Sampling of Hand-Object Interactions (CVPR 2024 · similarity 0.87)
- HO-Cap: A Capture System and Dataset for 3D Reconstruction and Pose Tracking of Hand-Object Interaction (NeurIPS 2025 · similarity 0.87)
- Single-view Image to Novel-view Generation for Hand-Object Interactions (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.