Research map / 3D vision / Human pose estimation
Transformer-Based Unified Recognition of Two Hands Manipulating Objects
Hoseong Cho, Chanwoo Kim, Jihyeon Kim, Seongyeong Lee, Elkhan Ismayilzada, Seungryul Baek
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add reference point refinement to the hand-object pose estimator (ablation) | fHOP pose estimator · H2O and FPHA test | mean end-point error (mm): Left hand -3.5mm, right hand -5.3mm, object -5.8mm on H2O; right hand -9.8mm, object -19.8mm on FPHA | Improved |
| add contact map to mesh vertex input of the interaction recognizer (ablation) | fIA interaction recognizer · H2O test | top-1 accuracy (%): 86.8 to 90.9 (+4.1) | Improved |
| add contact map to mesh vertex input of the interaction recognizer (ablation) | fIA interaction recognizer · FPHA test | top-1 accuracy (%): 96.8 to 98.4 (+1.6, 2000 samples only) | No clear effect |
| use mesh vertices and contact maps instead of raw hand poses for interaction recognition (ablation) | fIA interaction recognizer · H2O test | top-1 accuracy (%): 89.3 to 90.9 (+1.6) | Improved |
| use mesh vertices and contact maps instead of raw hand poses for interaction recognition (ablation) | fIA interaction recognizer · FPHA test | top-1 accuracy (%): 91.9 to 98.4 (+6.5) with full contact; pose-only is higher than mesh without contact (96.8 vs 91.9 comparison shows gain | Mixed |
| reduce contact map neighbor search to 10 nearest vertices instead of all-pairs distance (ablation) | contact map generation · H2O test | top-1 accuracy (%): 90.9 to 82.2 (-8.7) | Hurt |
| replace Transformer pose estimator with CNN pose estimator (Kwon et al.) in interaction pipeline (ablation) | pose estimation and fIA pipeline · H2O test | top-1 accuracy (%): 90.9 to 80.9 (-10.0) | Hurt |
| replace Transformer interaction recognizer with LSTM (ablation) | fIA interaction recognizer · H2O test | top-1 accuracy (%): 90.9 to 82.6 (-8.3) | Hurt |
| Transformer-based unified framework (H2OTR) vs prior methods on pose estimation | H2OTR · H2O and FPHA test | mean end-point error (mm): H2O right hand 25.8 vs 37.2 (Kwon et al.); FPHA object 21.0 vs 24.9 (Tekin et al.) | Improved |
| Transformer-based interaction recognition with contact maps vs prior interaction recognizers | H2OTR · H2O test | top-1 accuracy (%): 90.9 vs 86.4 (Wen et al.) | Improved |
Closest papers by meaning
- H+O: Unified Egocentric Recognition of 3D Hand-Object Poses and Interactions (CVPR 2019 · similarity 0.92)
- H2O: Two Hands Manipulating Objects for First Person Interaction Recognition (ICCV 2021 · similarity 0.91)
- Hierarchical Temporal Transformer for 3D Hand Pose Estimation and Action Recognition From Egocentric RGB Videos (CVPR 2023 · similarity 0.90)
- Keypoint Transformer: Solving Joint Identification in Challenging Hands and Object Interactions for Accurate 3D Pose Estimation (CVPR 2022 · similarity 0.89)
- Modeling Fine-Grained Hand-Object Dynamics for Egocentric Video Representation Learning (ICLR 2025 · similarity 0.89)
- Collaborative Learning for 3D Hand-Object Reconstruction and Compositional Action Recognition from Egocentric RGB Videos Using Superquadrics (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
