Research map / Reinforcement learning / Visual and representation RL
ManipTrans: Efficient Dexterous Bimanual Manipulation Transfer via Residual Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Two-stage PPO framework that pre-trains a generalist hand-motion imitator and then fine-tunes a residual module for object interaction, transferring human bimanual MoCap to simulated dexterous hands and producing the DEXMANIPNET dataset (3.3K episodes).
Problem
Transferring human hand-object MoCap manipulation sequences, especially complex bimanual tasks, to dexterous robotic hands in simulation with physical plausibility and high precision, where direct retargeting and RL from scratch are inaccurate or slow.
Approach
Decouple hand-motion imitation (pre-trained on hand-only data as a generalist imitator) from object interaction by learning a residual action policy on top of the frozen imitator, with interaction-aware state (contact forces, BPS object shape), gravity/friction relaxation curriculum, and no task-specific reward engineering.
Key results
- MANIPTRANS reaches 58.1% single-hand and 39.5% bimanual success rate, versus 47.8%/13.9% for Retarget + Residual and 34.3%/12.1% for RL-Only.
- Training an unseen 60-frame single-hand trajectory takes about 15 minutes on a personal RTX 4090, versus about 40 hours for QuasiSim's optimization.
- Diffusion Policy with Transformer backbone reaches 14.69% and DP-UNet 18.44% success on the bottle rearrangement benchmark, with all baselines low.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the two-stage MANIPTRANS transfer framework instead of retargeting plus residual learning | Dexterous hand transfer · OakInk-V2 validation | object rotation error: 11.58 -> 8.60 | Helped |
| Use the two-stage MANIPTRANS transfer framework instead of retargeting plus residual learning | Dexterous hand transfer · OakInk-V2 validation | object translation error: 0.79 -> 0.49 | Helped |
| Use the two-stage MANIPTRANS transfer framework instead of retargeting plus residual learning | Dexterous hand transfer · OakInk-V2 validation | success rate: 47.8 / 13.9 -> 58.1 / 39.5 | Helped |
| Add contact force to the reward (ablation) | MANIPTRANS · Contact-rich tasks | success rate | Helped |
| Add contact force to the observation (ablation) | MANIPTRANS · Contact-rich tasks | training convergence | Helped |
| Add contact-force termination when a firm grasp is expected (ablation) | MANIPTRANS · Contact-rich tasks | overall convergence speed | Helped |
| Relax gravity early in training (ablation) | MANIPTRANS · Complex bimanual tasks | success rate | Helped |
| Increase friction early in training (ablation) | MANIPTRANS · Complex bimanual tasks | success rate | Helped |
| Relax the imitation and object-alignment thresholds early in training (ablation) | MANIPTRANS · Complex bimanual tasks | training convergence | Helped |
Surprising findings
- Retarget-Only baseline is nearly infeasible (0% bimanual success), showing that direct pose retargeting fails for bimanual tasks.
- All imitation learning policies trained on DEXMANIPNET perform poorly (max 18.44% success), indicating the dataset is challenging.
Limitations
- Some MoCap sequences with excessive interaction-pose noise cannot be transferred effectively.
- Inaccurate object models for simulation, especially articulated objects, hurt transfer.
- Real-world deployment is demonstrated qualitatively only and limited to replaying trajectories.
Experimental setup
Datasets: OakInk-V2 FAVOR GRAB ARCTIC
Benchmarks: OakInk-V2 validation set (~80 episodes) bottle rearrangement policy benchmark on DEXMANIPNET
Metrics: object rotation error object translation error mean per-joint position error mean per-fingertip position error success rate
Baselines: RL-Only Retarget + Residual Retarget-Only QuasiSim IBC BET DP-UNet DP-Trans
Builds on: PPO residual policy learning QuasiSim BPS
Closest papers by meaning
- DexFlyWheel: A Scalable and Self-improving Data Generation Framework for Dexterous Manipulation (NeurIPS 2025 · similarity 0.91)
- Scaffolding Dexterous Manipulation with Vision-Language Models (NeurIPS 2025 · similarity 0.91)
- UniGraspTransformer: Simplified Policy Distillation for Scalable Dexterous Robotic Grasping (CVPR 2025 · similarity 0.89)
- HandEdit: A Unified Benchmark for Egocentric Human-to-Robot Dexterous Hand Image Editing (NeurIPS 2026 · similarity 0.89)
- DexTrack: Towards Generalizable Neural Tracking Control for Dexterous Manipulation from Human References (ICLR 2025 · similarity 0.89)
- HumanoidGen: Data Generation for Bimanual Dexterous Manipulation via LLM Reasoning (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.