Research map / 3D vision / Human pose estimation
Learning Physics-Based Full-Body Human Reaching and Grasping from Brief Walking References
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Generates physics-based full-body reaching and grasping motions from only ~30s of walking MoCap by combining an ASE-style skill space with actively generated kinematic grasp interpolations and a shallow-layer feature alignment reward.
Problem
Physics-based full-body human reaching and grasping motion synthesis is limited by scarce and biased MoCap data for reaching/grasping tasks, so existing data-driven methods struggle to generalize to unseen scenes and objects.
Approach
Use brief walking MoCap as the only real reference to learn a latent motion skill space (ASE-style), then actively augment it with kinematically generated grasp-interpolated motions chosen by a performance-based score, and regularize shallow critic features to align with walking features (local feature alignment via Mahalanobis reward).
Key results
- Ours reaches 99.8% SR(Grasp) and 88.8% SR(Goal) in simple scenes, outperforming AMP (85.3%/58.1%) and PSE (99.6%/85.5%).
- In complex scenes with unseen objects, Ours achieves 69.7% SR(Grasp) and 55.8% SR(Goal), comparable to the Oracle Policy (69.7%/53.4%).
- Active-Both data selection at 20% ratio yields 92.4% SR(Grasp) and 69.1% SR(Goal) versus 92.1%/64.1% for random selection at 20%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use Active-Both data selection instead of random selection at 20% data (ablation) | reaching and grasping policy · Simple Scenes | SR(Goal): 64.1% -> 69.1% | Helped |
| combine success-rate and discriminator-score estimates to select generated data (ablation) | active data augmentation · Simple Scenes | SR(Goal): 64.2% -> 69.1% | Helped |
| combine success-rate and discriminator-score estimates to select generated data (ablation) | active data augmentation · Simple Scenes | SR(Grasp): 95.2% -> 92.4% | Mixed |
| increase generated-data ratio from 5% to 20% with Active-Both (ablation) | active data augmentation · Simple Scenes | SR(Goal): 36.3% -> 69.1% | Helped |
| align f0 and f1 features instead of using no feature alignment (ablation) | reaching and grasping policy · Simple Scenes | SR(Goal): 69.1% -> 88.8% | Helped |
| align f0 features without aligning f1 (ablation) | reaching and grasping policy · Simple Scenes | SR(Goal): 69.1% -> 84.6% | Helped |
| add f1 alignment to f0 alignment (ablation) | reaching and grasping policy · Simple Scenes | SR(Goal): 84.6% -> 88.8% | Helped |
| also align f2 features in addition to f0 and f1 (ablation) | reaching and grasping policy · Simple Scenes | SR(Goal): 88.8% -> 47.5% | Hurt |
| align f0 and f1 features instead of using no alignment (ablation) | reaching and grasping policy · Simple Scenes | User(I): 1.17 -> 4.94 | Helped |
| compare Ours with PSE(2-Part) | reaching and grasping policy · Simple Scenes | SR(Goal): 85.5% -> 88.8% | Helped |
| compare Ours with PSE(5-Part) | reaching and grasping policy · Complex Scenes | SR(Goal): 42.6% -> 55.8% | Helped |
| compare Ours with Oracle Grasp Policy | reaching and grasping policy · Complex Scenes | SR(Grasp): 75.8% -> 69.7% | Hurt |
Surprising findings
- Higher added data ratios can degrade performance and lengthen training before convergence.
- Aligning the critic's shallow-layer features with only the f0 and f1 layers works best; adding deeper feature alignment (f2) drops success and user scores sharply.
Limitations
- Generated interpolated data lacks natural dynamics and introduces artifacts, requiring feature alignment to mitigate.
- Evaluation of motion naturalness relies on a user study and LLM judges rather than standard distribution metrics, which the authors argue are unreliable.
- Oracle Policy and some comparisons rely on real CIRCLE data, so the gap from walking-only data is not fully closed in complex scenes.
Experimental setup
Datasets: AMASS CMU MoCap CIRCLE UnidexGrasp object dataset
Benchmarks: simple scenes complex scenes
Metrics: SR(Grasp) SR(Goal) user preference (General Quality, Motion Issues) GPT-4o/Kimi scores FID (pilot study)
Baselines: Fullbody PPO ASE AMP PMP PSE Oracle Grasp Policy Oracle Policy WANDR Braun et al. Omnigrasp
Builds on: ASE AMP UnidexGrasp FLEX PPO
Closest papers by meaning
- DiffGrasp: Whole-Body Grasping Synthesis Guided by Object Motion Using a Diffusion Model (AAAI 2025 · similarity 0.82)
- Inferring Forces and Learning Human Utilities From Videos (CVPR 2016 · similarity 0.82)
- PhysPT: Physics-aware Pretrained Transformer for Estimating Human Dynamics from Monocular Videos (CVPR 2024 · similarity 0.82)
- Physical Plausibility-aware Trajectory Prediction via Locomotion Embodiment (CVPR 2025 · similarity 0.80)
- InterMimic: Towards Universal Whole-Body Control for Physics-Based Human-Object Interactions (CVPR 2025 · similarity 0.80)
- BOTH2Hands: Inferring 3D Hands from Both Text Prompts and Body Dynamics (CVPR 2024 · similarity 0.79)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.