Research map / 3D vision / Human pose estimation

ComPose: When to Trust Hands for Object Pose Tracking

NeurIPS 2026 · 3D vision › Human pose estimation · approach: Depth-based hand pose regression · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes ComPose, a hand-aware 6DoF object pose tracker from RGB video that adaptively fuses ICP-based object rotation with hand-joint Procrustes rotation (learned joint weights and blending) plus a learned translation residual, and transfers trajectories to a Franka robot.

Problem

Object pose tracking from monocular RGB video under severe hand occlusion and geometric ambiguity (e.g., symmetric objects), where prior methods rely on depth, CAD models, or templates.

Approach

Treat the hand as a complementary cue: a learned module predicts per-joint weights for weighted Procrustes hand rotation, a blending coefficient alpha that SLERPs object and hand relative rotations, and a residual translation over the object-point-cloud centroid, trained with cumulative-rotation consistency and translation-smoothness losses.

Key results

• On DexYCB, ComPose reaches RRE 1.5 deg, RTE 0.6, ATE 3.0 and TCC_R 0.55, outperforming MegaPose, FoundPose, FreePose and UniHOPE on the relative and temporal metrics.
• On HO-3D v2 ComPose achieves RRE 2.4 deg and TCC_R 0.41, the best relative/temporal numbers, but ARE (49.2) is worse than FoundPose (46.0) and ATE (2.7) ties FreePose.
• Ablation: object-only ICP gives ARE 35.02 and RRE 1.97 on DexYCB versus full ComPose ARE 23.72 with VGGT and 19.88 with HaWoR, showing that combining object and hand cues helps.

Surprising findings

• Using VGGT instead of Flow3R gives lower RRE (0.81) on DexYCB but worse translation metrics (ATE 4.52), a mixed trade-off.
• Object-only ICP on DexYCB already reaches low RTE (0.6) and comparable TCC_T, so the hand cue mainly improves rotation.

Limitations

• Fails when both object geometry and hand cues are unreliable, such as severe occlusion, motion blur, or degraded images.
• Struggles with unseen grasp distributions where it may not identify informative joints.
• Anchoring reduces but does not eliminate cumulative rotation drift over long sequences.

Experimental setup

Datasets: DexYCB HOI4D HO-3D v2 OakInk-v1

Benchmarks: DexYCB S3 test HO-3D v2 OakInk-v1

Metrics: RRE RTE ARE ATE TCC_R TCC_T inference time

Baselines: MegaPose FoundPose FreePose UniHOPE

Builds on: Flow3R WiLoR SAM3 DINOv2 SAM3D-Object FreePose

Closest papers by meaning

• Hand-held Object Reconstruction from RGB Video with Dynamic Interaction (CVPR 2025 · similarity 0.90)
• StableHand: Quality-Aware Flow Matching for World-Space Dual-Hand Motion Estimation from Egocentric Video (NeurIPS 2026 · similarity 0.90)
• ManiVideo: Generating Hand-Object Manipulation Video with Dexterous and Generalizable Grasping (CVPR 2025 · similarity 0.89)
• Hand-Centric Motion Refinement for 3D Hand-Object Interaction via Hierarchical Spatial-Temporal Modeling (AAAI 2024 · similarity 0.89)
• HOLD: Category-agnostic 3D Reconstruction of Interacting Hands and Objects from Video (CVPR 2024 · similarity 0.89)
• HO-Cap: A Capture System and Dataset for 3D Reconstruction and Pose Tracking of Hand-Object Interaction (NeurIPS 2025 · similarity 0.88)