Research map / Reinforcement learning / Visual and representation RL
World Models for Learning Dexterous Hand-Object Interactions from Human Videos
Raktim Goswami, Amir Bar, David Fan, Tsung-Yen Yang, Gaoyue Zhou, Prashanth Krishnamurthy, Michael Rabbat, Farshad Khorrami, Yann LeCun
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add hand consistency loss (ablation) | DexWM · EgoDex open-loop | PCK@20 at 4s: 26 -> 60 | Improved |
| add hand consistency loss (ablation) | DexWM · EgoDex open-loop | embedding L2 error at 4s: 0.85 -> 0.66 | Improved |
| add EgoDex human video on top of DROID (ablation) | DexWM · RoboCasa Lift open-loop | PCK@20 at 4s: 2 -> 7 | Improved |
| add DROID to EgoDex training (ablation) | DexWM · EgoDex open-loop | PCK@20 at 4s: 60 -> 60 | No clear effect |
| human-video pretraining before robot fine-tuning (ablation) | DexWM · RoboCasa grasp | success rate (%): 14 -> 58 | Improved |
| human-video pretraining before robot fine-tuning (ablation) | DexWM · RoboCasa reach | success rate (%): 18 -> 72 | Improved |
| pretrain Diffusion Policy on the same human data (ablation) | Diffusion Policy · RoboCasa avg | success rate (%): 4% vs DexWM 53% | No clear effect |
| use world model with MPC instead of goal-conditioned diffusion policy | DexWM · real robot grasp | success rate (%): 0 -> 83 | Improved |
| increase predictor size S to XL (31M to 456M) (ablation) | DexWM · open-loop EgoDex/RoboCasa | embedding error and PCK@20 | Improved |
| swap DINOv2 for DINOv3, Web-SSL, V-JEPA 2 or SigLIP 2 (ablation) | DexWM · RoboCasa sim tasks | success rate | Mixed |
| add keypoint cost to latent state planning cost (ablation) | DexWM · robot planning | success rate | Improved |
| Lift-initialized exploration instead of random-target exploration (ablation) | DexWM · RoboCasa avg | success rate (%): 49 -> 53 | Improved |
Closest papers by meaning
- DexNDM: Closing the Reality Gap for Dexterous In-Hand Rotation via Joint-Wise Neural Dynamics Model (ICLR 2026 · similarity 0.92)
- Controlling the World by Sleight of Hand (ECCV 2024 · similarity 0.90)
- Learning Object-Centric Motion Priors from Human for Robotic Dexterous Manipulation (AAAI 2026 · similarity 0.90)
- Scaffolding Dexterous Manipulation with Vision-Language Models (NeurIPS 2025 · similarity 0.90)
- ManipTrans: Efficient Dexterous Bimanual Manipulation Transfer via Residual Learning (CVPR 2025 · similarity 0.89)
- HandEdit: A Unified Benchmark for Egocentric Human-to-Robot Dexterous Hand Image Editing (NeurIPS 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
