Research map / Detection and segmentation / Open-set and few-shot detection
GEARS: Local Geometry-aware Hand-object Interaction Synthesis
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes GEARS, a joint-centered point-based local geometry sensor plus spatio-temporal attention network that synthesizes hand motion sequences from hand and object trajectories, with heuristic augmentation of static grasps.
Problem
Generating realistic hand motion sequences interacting with objects generalizes poorly across object categories, shapes and sizes, due to limited sensor expressiveness and scarce dynamic hand-object training data.
Approach
A joint-local point-based geometry sensor that samples object surface points near each hand joint and expresses them in a canonical MANO joint frame, processed by a shared PointNet and refined with spatial and temporal self-attention; static grasps from ObMan are turned into synthetic sequences by interpolation.
Key results
- On GRAB, GEARS reaches MPJPE 7.24 mm, PD 4.36 mm, IV 2.24 cm3 and C-IoU 22.7%, beating GRIP (7.71 mm, 19.9%) and TOCH (8.18 mm, 20.1%).
- On InterCap, GEARS achieves PD 7.44 mm and IV 5.21 cm3, lower than GRIP (7.92, 5.68) and ManipNet (8.22, 6.15).
- Ablation shows removing the Joint Displacement Network raises MPJPE from 7.24 to 9.63 mm and drops C-IoU from 22.7% to 13.2%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use GEARS instead of GRIP | hand-object interaction synthesis · GRAB | MPJPE (mm): 7.71 -> 7.24 | Helped |
| use GEARS instead of GRIP | hand-object interaction synthesis · GRAB | PD (mm): 4.80 -> 4.36 | Helped |
| use GEARS instead of GRIP | hand-object interaction synthesis · GRAB | IV (cm3): 2.51 -> 2.24 | Helped |
| use GEARS instead of GRIP | hand-object interaction synthesis · GRAB | C-IoU (%): 19.9 -> 22.7 | Helped |
| use GEARS instead of GRIP | hand-object interaction synthesis · InterCap | PD (mm): 7.92 -> 7.44 | Helped |
| use GEARS instead of GRIP | hand-object interaction synthesis · InterCap | IV (cm3): 5.68 -> 5.21 | Helped |
| add a joint-local sensor with radius 0 rather than the full method's setting (ablation) | GEARS · GRAB | MPJPE (mm): 7.24 -> 9.34 | Hurt |
| add a joint-local sensor with radius 0 rather than the full method's setting (ablation) | GEARS · GRAB | C-IoU (%): 22.7 -> 14.8 | Hurt |
| add the Joint Displacement Network (ablation) | GEARS · GRAB | MPJPE (mm): 9.63 -> 7.24 | Helped |
| add the Joint Displacement Network (ablation) | GEARS · GRAB | C-IoU (%): 13.2 -> 22.7 | Helped |
| use spatio-temporal attention instead of fully-connected layers (ablation) | GEARS · GRAB | MPJPE (mm): 7.85 -> 7.24 | Helped |
| add synthetic training sequences (ablation) | GEARS · GRAB | C-IoU (%): 20.6 -> 22.7 | Helped |
Surprising findings
- Iterative refinement at inference (feeding Joint Displacement Network output back as input) gives no benefit, attributed to distribution shift.
- Higher MPJPE does not imply worse grasp quality: TOCH has higher MPJPE than GRIP but higher contact IoU.
Limitations
- Synthetic sequences from static grasps may contain hand-object intersections that are filtered by a threshold rather than avoided.
- Input hand trajectories on InterCap are noisy, and the GT annotations are not accurate enough for full evaluation, so only PD and IV are reported there.
- Iterative inference refinement did not help, suggesting train-test distribution mismatch.
Experimental setup
Datasets: GRAB InterCap ObMan
Benchmarks: GRAB InterCap
Metrics: MPJPE penetration depth intersection volume contact IoU
Baselines: TOCH ManipNet GRIP
Builds on: ManipNet GRIP TOCH PointNet MANO
Closest papers by meaning
- Hand-Centric Motion Refinement for 3D Hand-Object Interaction via Hierarchical Spatial-Temporal Modeling (AAAI 2024 · similarity 0.91)
- LatentHOI: On the Generalizable Hand Object Motion Generation with Latent Hand Diffusion. (CVPR 2025 · similarity 0.89)
- MEgoHand: Multimodal Egocentric Hand-Object Interaction Motion Generation (NeurIPS 2025 · similarity 0.89)
- How Do I Do That? Synthesizing 3D Hand Motion and Contacts for Everyday Interactions (CVPR 2025 · similarity 0.89)
- Text2HOI: Text-guided 3D Motion Generation for Hand-Object Interaction (CVPR 2024 · similarity 0.89)
- DiffGrasp: Whole-Body Grasping Synthesis Guided by Object Motion Using a Diffusion Model (AAAI 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.