Research map / 3D vision / Human pose estimation
HO-Cap: A Capture System and Dataset for 3D Reconstruction and Pose Tracking of Hand-Object Interaction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces HO-Cap, a multi-camera RGB-D plus HoloLens capture system and semi-automatic annotation pipeline (pretrained vision models with SDF-based refinement), and releases a 64-video hand-object interaction dataset with baseline results.
Problem
Obtaining scalable 3D shape and pose annotations for hands and objects in videos is costly, since mocap and 3D scanners are expensive and manual labeling is slow; existing semi-automatic systems need per-object training or known models.
Approach
A markerless multi-view RGB-D capture setup with a HoloLens headset, and an annotation pipeline that chains pretrained models (BundleSDF, SAM2, FoundationPose, MediaPipe) with multi-view RANSAC pose fusion, SDF-based pose refinement, and joint hand-object optimization, requiring no domain-specific training.
Key results
- Annotation error after joint refinement is about 3.4 to 4.6 pixels for objects and hands, reduced from initial poses (3.67 object, 5.31 left hand, 5.16 right hand).
- HaMeR reaches 28.9 mm MPJPE and 88.5% PCK(0.15) on HO-Cap hand pose, versus 78.7 mm for A2J-Transformer.
- FoundationPose achieves 89.3% ADD AUC and 95.7% ADD-S AUC on novel objects, versus 67.1% and 83.0% for MegaPose.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use HaMeR instead of A2J-Transformer | hand pose estimation · HO-Cap | MPJPE (mm): 78.7 -> 28.9 | Helped |
| Use HaMeR instead of A2J-Transformer | hand pose estimation · HO-Cap | PCK(0.2): 50.5 -> 91.4 | Helped |
| Train the detector on HO-Cap data using RT-DETR* rather than novel-object baseline CNOS | object detection · HO-Cap objects | AP: 25.3 -> 75.9 | Helped |
| Use FoundationPose instead of MegaPose | novel object pose estimation · HO-Cap | ADD AUC (%): 67.1 -> 89.3 | Helped |
| Use FoundationPose instead of MegaPose | novel object pose estimation · HO-Cap | ADD-S AUC (%): 83.0 -> 95.7 | Helped |
| Add SDF-based pose refinement (ablation) | HO-Cap annotation pipeline · HO-Cap object poses | Pixel error (mean ± std): 3.67 (± 3.05) -> 3.50 (± 1.70) | Helped |
| Add SDF-based pose refinement (ablation) | HO-Cap annotation pipeline · HO-Cap left-hand poses | Pixel error (mean ± std): 5.31 (± 3.43) -> 4.58 (± 2.73) | Helped |
| Add SDF-based pose refinement (ablation) | HO-Cap annotation pipeline · HO-Cap right-hand poses | Pixel error (mean ± std): 5.16 (± 3.73) -> 3.83 (± 2.29) | Helped |
| Jointly refine hand and object poses after SDF refinement (ablation) | HO-Cap annotation pipeline · HO-Cap object poses | Pixel error (mean ± std): 3.50 (± 1.70) -> 3.42 (± 1.73) | Helped |
| Jointly refine hand and object poses after SDF refinement (ablation) | HO-Cap annotation pipeline · HO-Cap left-hand poses | Pixel error (mean ± std): 4.58 (± 2.73) -> 4.58 (± 2.72) | No clear effect |
| Jointly refine hand and object poses after SDF refinement (ablation) | HO-Cap annotation pipeline · HO-Cap right-hand poses | Pixel error (mean ± std): 3.83 (± 2.29) -> 3.82 (± 2.29) | Helped |
Surprising findings
- HaMeR's MPJPE of 28.9 mm is much higher than errors reported on other datasets, indicating HO-Cap is challenging due to hand-object occlusion.
Limitations
- Textureless and metallic objects cannot be reconstructed well by BundleSDF and were excluded.
- MediaPipe misses hand joints occasionally, causing videos with frequent failures to be dropped.
- Small or cylindrical objects held in hand yield poor rotation estimates and are excluded.
Experimental setup
Datasets: HO-Cap InterHand2.6M YCB Object Set
Benchmarks: HO-Cap
Metrics: PCK MPJPE AP AP50 AP75 ADD AUC ADD-S AUC pixel error
Baselines: A2J-Transformer HaMeR CNOS GroundingDINO YOLO11 RT-DETR MegaPose FoundationPose
Builds on: BundleSDF FoundationPose SAM2 MediaPipe MANO
Closest papers by meaning
- HOLD: Category-agnostic 3D Reconstruction of Interacting Hands and Objects from Video (CVPR 2024 · similarity 0.89)
- InterAct: Advancing Large-Scale Versatile 3D Human-Object Interaction Generation (CVPR 2025 · similarity 0.89)
- HOT3D: Hand and Object Tracking in 3D from Egocentric Multi-View Videos (CVPR 2025 · similarity 0.88)
- HOI-Swap: Swapping Objects in Videos with Hand-Object Interaction Awareness (NeurIPS 2024 · similarity 0.88)
- HOIST-Former: Hand-held Objects Identification Segmentation and Tracking in the Wild (CVPR 2024 · similarity 0.88)
- HOIDiffusion: Generating Realistic 3D Hand-Object Interaction Data (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.