Research map / Reinforcement learning / Visual and representation RL
Cross-Embodiment Dexterous Grasping with Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Trains a single vision-based RL policy (CrossDex) that grasps YCB objects across four dexterous hands by acting in a MANO eigengrasp space with neural retargeting and a unified fingertip/palm observation space.
Problem
Learning one universal dexterous grasping policy that controls diverse robotic hands with different DoFs, finger counts and morphologies, and generalizes to unseen hands.
Approach
Use human-hand eigengrasps (PCA over MANO poses from GRAB) as a shared action space, convert them to each hand's joint targets via retargeting networks (DexPilot-trained MLPs), and observe only fingertip and palm positions so proprioception is embodiment-agnostic; distill per-object state-based PPO teachers into a vision policy with DAgger.
Key results
- CrossDex vision-based policy reaches 0.800 average success on the four training hands, versus 0.782 for the best-performing baseline MT-Raw-OA's vision policy (state-based MT-Raw-OA is 0.914).
- On unseen hands, CrossDex vision policy achieves 0.352 zero-shot success, versus 0.210 for MT-Raw-A.
- Finetuning on unseen LEAP Hand, CrossDex multi-task vision policy reaches 0.643 on YCB vs 0.436 for No-Pretrain, and 0.740 on 55 GRAB objects vs 0.313 for No-Pretrain.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use CrossDex's unified eigengrasp actions and shared observations instead of raw hand actions and observations | CrossDex · YCB objects, training hands (Vision) | success rate: 0.779 -> 0.800 | Helped |
| Use CrossDex's unified eigengrasp actions and shared observations instead of raw hand actions and observations | CrossDex · YCB objects, unseen hands (Vision) | success rate: 0.210 -> 0.352 | Helped |
| Pretrain CrossDex before finetuning on the unseen LEAP Hand | Vision-based policy on YCB, multi-task · YCB, multi-task (Vision), unseen LEAP Hand | success rate: 0.436$\pm$0.159 -> \textbf{0.643}$\pm$0.009 | Helped |
| Pretrain CrossDex before finetuning on unseen objects | Vision-based policy on GRAB objects, multi-task · GRAB, 55 objects (Vision), unseen LEAP Hand | success rate: 0.655$\pm$0.006 -> \textbf{0.740}$\pm$0.009 | Helped |
| Add mount-position randomization (ablation) | CrossDex without embodiment observations · YCB objects, unseen hands (Vision) | success rate: 0.209 -> 0.352 | Helped |
| Add mount-position randomization (ablation) | CrossDex without embodiment observations · YCB objects, training hands (Vision) | success rate: 0.735 -> 0.800 | Helped |
| Add embodiment-specific observations (ablation) | CrossDex with mount-position randomization · YCB objects, unseen hands (Vision) | success rate: 0.352 -> 0.187 | Hurt |
| Add embodiment-specific observations (ablation) | CrossDex without mount-position randomization · YCB objects, unseen hands (Vision) | success rate: 0.209 -> 0.146 | Hurt |
| Use 1 eigengrasp instead of a larger eigengrasp set (ablation) | CrossDex · YCB objects, five-object ablation | training performance | Hurt |
| Use 45-dimensional MANO axis-angle actions instead of eigengrasp actions (ablation) | CrossDex · Zero-shot adaptation to unseen hands | success rate | Hurt |
| Vary the number of eigengrasps from 1 to 36 (ablation) | CrossDex · YCB objects, five-object ablation | training and test performance | Mixed |
| Replace DexPilot with another optimization-based retargeting method (ablation) | CrossDex state-based policy · Five randomly chosen YCB objects | training and test performance | No clear effect |
Surprising findings
- Embodiment-specific observations (one-hot robot labels and randomized mount positions) hurt generalization to unseen hands, reducing unseen-hand state success from 0.259 to 0.044 when randomization is also applied without them dropped.
- Raw action space baselines (MT-Raw-A) still generalize to unseen hands partly because URDFs were manually aligned in joint ordering and motion direction.
Limitations
- Only four dexterous hands used for training due to limited hand model access.
- Evaluated only on grasping tasks, not in-hand reorientation, handover or functional grasping.
- Zero-shot success on unseen hands (0.352 vision) remains much lower than on training hands (0.800).
Experimental setup
Datasets: YCB GRAB
Benchmarks: YCB objects (45) in IsaacGym across training and unseen hands GRAB unseen objects (55) finetuning
Metrics: grasp success rate
Baselines: MT-Raw-OA MT-Raw-A MT-Raw-O No-Pretrain
Builds on: PPO DAgger UniDexGrasp DexPilot eigengrasps UniDexGrasp++
Closest papers by meaning
- Scaffolding Dexterous Manipulation with Vision-Language Models (NeurIPS 2025 · similarity 0.91)
- ManipTrans: Efficient Dexterous Bimanual Manipulation Transfer via Residual Learning (CVPR 2025 · similarity 0.89)
- DexGrasp Anything: Towards Universal Robotic Dexterous Grasping with Physics Awareness (CVPR 2025 · similarity 0.88)
- Multi-fingered Hand Grasps with Visuo-Tactile Fusion via Multi-Agent Deep Reinforcement Learning (AAAI 2025 · similarity 0.88)
- UniGraspTransformer: Simplified Policy Distillation for Scalable Dexterous Robotic Grasping (CVPR 2025 · similarity 0.88)
- HandEdit: A Unified Benchmark for Egocentric Human-to-Robot Dexterous Hand Image Editing (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.