Research map / 3D vision / 3D generation
Hand-Centric Motion Refinement for 3D Hand-Object Interaction via Hierarchical Spatial-Temporal Modeling
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a hand-centric representation and a hierarchical spatial-temporal network (HST-Net) with post-optimization via MANO to refine jittery, perturbed 3D hand-object interaction motion sequences.
Problem
Coarse 3D hand motion from grasp tracking or interaction synthesis is noisy and jittery with inconsistent contact and interpenetration; refining such hand-object interaction sequences is the target.
Approach
Replaces TOCH's object-centric ray-based representation with a hand-centric one (per-hand-vertex displacement to its closest object vertex), and models spatial structure hierarchically (local-to-global point features) and temporal structure with long-term then short-term transformers, followed by MANO-based smoothing.
Key results
- On GRAB-B (balanced perturbation), Ours reaches MPJPE 7.17 mm and C-IoU 26.36% versus TOCH's 10.3 mm and 23.10%.
- On HO3D, Ours achieves MPJPE 9.18 mm vs TOCH 9.32 mm and IV 4.52 cm^3 vs 4.66.
- On the 30cm longer-distance setting, Ours reaches MPJPE 7.80 mm vs TOCH 14.11 mm.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the proposed method instead of TOCH | Hand-object motion refinement · GRAB-B (0.01 & 0.3) | MPJPE (mm): 10.3 -> 7.17 | Helped |
| Use the proposed method instead of TOCH | Hand-object motion refinement · GRAB-B (0.01 & 0.3) | MPVPE (mm): 12.1 -> 8.09 | Helped |
| Use the proposed method instead of TOCH | Hand-object motion refinement · GRAB-B (0.01 & 0.3) | IV (cm³): 1.78 -> 1.69 | Helped |
| Use the proposed method instead of TOCH | Hand-object motion refinement · GRAB-B (0.01 & 0.3) | C-IoU (%): 23.10* -> 26.36 | Helped |
| Use the proposed method instead of TOCH | Hand-object motion refinement · HO3D | MPJPE: 9.32 -> 9.18 | Helped |
| Use the proposed method instead of TOCH | Hand-object motion refinement · HO3D | MPVPE: 9.28 -> 9.21 | Helped |
| Use the proposed method instead of TOCH | Hand-object motion refinement · HO3D | IV: 4.66 -> 4.52 | Helped |
| Use closest-object vertices rather than hand vertices in the interaction representation (ablation) | Hand-centric representation · GRAB-B | MPJPE: 7.89 -> 7.17 | Helped |
| Use closest-object vertices rather than the object mass center in the interaction representation (ablation) | Hand-centric representation · GRAB-B | MPJPE: 7.89 -> 7.17 | Helped |
| Use closest-object vertices rather than bounding-box vertices in the interaction representation (ablation) | Hand-centric representation · GRAB-B | MPJPE: 8.02 -> 7.17 | Helped |
| Add hierarchical spatial encoding rather than four PointNet layers (ablation) | HST-Net · GRAB-B | MPJPE: 7.35 -> 7.17 | Helped |
| Use the hierarchical temporal design rather than swapping long-term and short-term transformers (ablation) | HST-Net · GRAB-B | MPJPE: 8.85 -> 7.17 | Helped |
Surprising findings
- On GRAB-R (0.3), TOCH achieves lower intersection volume (1.52) than the proposed method (1.61), a marginal regression the authors acknowledge.
Limitations
- Assumes ground-truth object sequences are available during training and testing.
- Trained only on balanced perturbation (GRAB-B); evaluated on other perturbations and real tracking errors without retraining.
- Marginal inferior intersection volume on GRAB-R compared with TOCH.
Experimental setup
Datasets: GRAB HO3D
Benchmarks: GRAB HO3D
Metrics: MPJPE MPVPE IV C-IoU
Baselines: TOCH Hasson et al. tracking RefineNet ContactOpt
Builds on: TOCH PointNet MANO
Closest papers by meaning
- GEARS: Local Geometry-aware Hand-object Interaction Synthesis (CVPR 2024 · similarity 0.91)
- How Do I Do That? Synthesizing 3D Hand Motion and Contacts for Everyday Interactions (CVPR 2025 · similarity 0.90)
- LatentHOI: On the Generalizable Hand Object Motion Generation with Latent Hand Diffusion. (CVPR 2025 · similarity 0.90)
- MEgoHand: Multimodal Egocentric Hand-Object Interaction Motion Generation (NeurIPS 2025 · similarity 0.90)
- HandDiffuse: Generative Controllers for Two-Hand Interactions via Diffusion Models (AAAI 2025 · similarity 0.90)
- Text2HOI: Text-guided 3D Motion Generation for Hand-Object Interaction (CVPR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.