Research map / Detection and segmentation / Open-set and few-shot detection
Physics-Aware Hand-Object Interaction Denoising
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a physics-aware hand-object interaction denoising network trained with two learned differentiable physical losses (grasp credibility and manipulation feasibility) on top of a dual HOI representation.
Problem
Vision-based hand trackers produce physically implausible hand-object interaction sequences (penetration, erroneous contact, unrealistic manipulation) due to occlusion; existing denoisers are purely data-driven or optimization-based and not physically aware.
Approach
Learns two differentiable neural physical losses: a grasp-credibility predictor for deep penetration (penetration depth metric with hard and soft targets) and a manipulation-feasibility predictor based on force error and manipulation expense from friction-cone contact force optimization; these are used to train a denoising auto-encoder over a dual hand-object representation (hand keypoints plus ob
Key results
- On HO-3D, the method reduces intersection volume to 5.95 versus 7.24 for TOCH and 8.33 for Physical Interaction, with plausible rate 0.85 and MPJPE 10.7 vs 11.4 for the Hasson et al. tracker.
- On GRAB with synthetic noise (T-0.01, θ-0.3 training), MPJPE drops from 16.01 to 6.91 and plausible rate rises from 0.42 to 0.95 when training on the θ-0.3 noise setting.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add manipulation feasibility loss (ablation) | Ours on GRAB · GRAB | plausible rate: 0.82 -> 0.91 | Helped |
| add grasp credibility loss (ablation) | Ours on GRAB · GRAB | IV: 2.43 -> 1.13 | Helped |
| add grasp credibility loss (ablation) | Ours on GRAB · GRAB | plausible rate: 0.81 -> 0.91 | Helped |
| add manipulation feasibility loss (ablation) | Ours on GRAB · GRAB | IV: 0.96 -> 1.13 | Hurt |
| train grasp credibility loss with hard and soft targets instead of hard target only (ablation) | Grasp credibility loss · Perturbed GRAB test split | F-score: 0.85 -> 0.93 | Helped |
| train grasp credibility loss with hard and soft targets instead of hard target only (ablation) | Grasp credibility loss · GRAB | IV: 1.52 -> 1.13 | Helped |
| train manipulation feasibility loss with hard and soft targets instead of hard target only (ablation) | Manipulation feasibility loss · GRAB | plausible rate: 0.84 -> 0.91 | Helped |
| train manipulation feasibility loss with hard and soft targets instead of hard target only (ablation) | Manipulation feasibility loss · Perturbed GRAB test split | F-score: 0.90 -> 0.88 | Hurt |
| refine Hasson et al. tracker outputs with Ours instead of Hasson et al. | HO-3D hand tracking · HO-3D test split | plausible rate: 0.71 -> 0.85 | Helped |
| refine TOCH outputs with Ours instead of TOCH | HO-3D hand tracking · HO-3D test split | IV: 7.24 -> 5.95 | Helped |
| refine Physical Interaction outputs with Ours instead of Physical Interaction | HO-3D hand tracking · HO-3D test split | plausible rate: 0.83 -> 0.85 | No clear effect |
Surprising findings
- Removing the grasp credibility loss increases contact and manipulation feasibility but at the cost of more severe penetration, showing the two losses trade off against each other.
- Removing the manipulation feasibility loss yields the lowest IV and PD but more frequent hovering objects.
Limitations
- Assumes accurate object trajectory and only denoises the hand poses.
- Friction coefficient fixed at 0.8 and object mass set to a constant, so force metrics are relative.
- Only single-hand, single-rigid-object sequences are considered.
Experimental setup
Datasets: GRAB HO-3D
Benchmarks: GRAB HO-3D
Metrics: MPJPE MPVPE contact IoU intersection volume penetration depth plausible rate F-score
Baselines: TOCH Physical Interaction Hasson et al. hand tracker GT
Builds on: TOCH PointNet Physical Interaction (Han et al.) MANO
Closest papers by meaning
- ComPose: When to Trust Hands for Object Pose Tracking (NeurIPS 2026 · similarity 0.87)
- DiffGrasp: Whole-Body Grasping Synthesis Guided by Object Motion Using a Diffusion Model (AAAI 2025 · similarity 0.87)
- StableHand: Quality-Aware Flow Matching for World-Space Dual-Hand Motion Estimation from Egocentric Video (NeurIPS 2026 · similarity 0.86)
- GeneOH Diffusion: Towards Generalizable Hand-Object Interaction Denoising via Denoising Diffusion (ICLR 2024 · similarity 0.86)
- Hand-Centric Motion Refinement for 3D Hand-Object Interaction via Hierarchical Spatial-Temporal Modeling (AAAI 2024 · similarity 0.86)
- LatentHOI: On the Generalizable Hand Object Motion Generation with Latent Hand Diffusion. (CVPR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.