Research map / Detection and segmentation / Open-set and few-shot detection
Omnigrasp: Grasping Diverse Objects with Simulated Humanoids
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Omnigrasp trains a dexterous, full-body simulated SMPL-X humanoid with RL in a PULSE-X latent motion space to grasp over 1200 objects and follow randomly generated 3D object trajectories, guided by pre-grasp rewards.
Problem
Controlling a simulated humanoid with dexterous hands to grasp diverse objects and carry them along diverse trajectories, where prior work uses disembodied hands, limited trajectories, or per-object policies.
Approach
Use a pretrained universal dexterous full-body-and-hand motion latent space (PULSE-X, distilled from a finger-enabled PHC-X imitator on a hand-augmented AMASS dataset) as the RL action space, with simple object-centric state and a stepwise pre-grasp reward, and a 3D random trajectory generator so no paired full-body object motion is needed.
Key results
- On GRAB cross-object test, Omnigrasp (generated trajectories) reaches 100% grasp success and 94.1% trajectory success, versus 79% grasp success for Braun et al. (who report no trajectory success).
- Omnigrasp successfully picks up 1272 of 1330 training OakInk objects, and a policy trained only on GRAB picks up more than 1000 OakInk objects without training on them.
- Removing the PULSE-X latent action space drops trajectory success from 94.1% to 33.6% on the GRAB cross-object test.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the PULSE-X latent motion representation as the action space (ablation) | Omnigrasp · GRAB-Goal-Test (Cross-Object) | trajectory success: 33.6% -> 94.1% | Helped |
| Add pre-grasp guidance reward (ablation) | Omnigrasp · GRAB-Goal-Test (Cross-Object) | grasp success: 77.1% -> 100% | Helped |
| Train PULSE-X on Dex-AMASS (ablation) | Omnigrasp · GRAB-Goal-Test (Cross-Object) | trajectory success: 77.3% -> 94.1% | Helped |
| Randomize the object's initial pose (ablation) | Omnigrasp · GRAB-Goal-Test (Cross-Object) | trajectory success: 79.9% -> 94.1% | Helped |
| Use hard-negative mining (ablation) | Omnigrasp · GRAB-Goal-Test (Cross-Object) | trajectory success: 71.6% -> 94.1% | Helped |
| Remove the object shape latent code (ablation) | Omnigrasp · GRAB-Goal-Test (Cross-Object) | trajectory success: 94.1% -> 93.2% | No clear effect |
| Replace the RNN policy with an MLP (ablation) | Omnigrasp · GRAB-Goal-Test (Cross-Object) | trajectory success: 94.1% -> 89.6% | Hurt |
| Provide ground-truth full-body pose as policy input (ablation) | Omnigrasp · GRAB-Goal-Test (Cross-Object) | trajectory success: 94.1% -> 77.8% | Hurt |
| Train with generated trajectories instead of MoCap trajectories | Omnigrasp · GRAB-Goal-Test (Cross-Object) | trajectory success: 84.8% -> 94.1% | Helped |
| Add uniform input noise at scale 0.01 | Omnigrasp · GRAB-Goal-Test (Cross-Object) | trajectory success: 94.1% -> 91.4% | Hurt |
Surprising findings
- Training only on GRAB generalizes to picking up more than 1000 OakInk objects without training on them.
- Adding uniform observation noise of 0.01 at test time only modestly degrades performance, even though the policy was not trained with noise.
Limitations
- Rotation error remains high relative to position, and no precise in-hand manipulation is supported.
- Trajectory-following success is limited because objects can be dropped or not picked up.
- Grasp types cannot be specified, and failures occur for objects that are too large or too small.
Experimental setup
Datasets: AMASS GRAB Re:InterHand OakInk OMOMO
Benchmarks: GRAB-Goal-Test (cross-object) GRAB-IMoS-Test (cross-subject) OakInk OMOMO
Metrics: grasp success rate trajectory success rate TTR position error rotation error acceleration error velocity error
Baselines: Braun et al. PHC AMP PPO-10B
Builds on: PULSE PHC PPO PACER AMP
Closest papers by meaning
- Humanoid Horizon: Extending Task Horizon in Whole-Body Loco-Manipulation via Parallel Training, Dynamic Starting, and Reward Gating (NeurIPS 2026 · similarity 0.88)
- AdaManip: Adaptive Articulated Object Manipulation Environments and Policy Learning (ICLR 2025 · similarity 0.86)
- Universal Humanoid Motion Representations for Physics-Based Control (ICLR 2024 · similarity 0.86)
- H-GAP: Humanoid Control with a Generalist Planner (ICLR 2024 · similarity 0.86)
- SUGAR: A Scalable Human-Video-Driven Generalizable Humanoid Loco-Manipulation Learning Framework (NeurIPS 2026 · similarity 0.86)
- DiffGrasp: Whole-Body Grasping Synthesis Guided by Object Motion Using a Diffusion Model (AAAI 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.