Research map / Reinforcement learning / Visual and representation RL

DexTrack: Towards Generalizable Neural Tracking Control for Dexterous Manipulation from Human References

ICLR 2025 · Reinforcement learning › Visual and representation RL · approach: Dexterous manipulation transfer · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Trains a generalizable neural dexterous-hand tracking controller from large curated robot tracking demonstrations, mined iteratively with a homotopy optimization scheme and a learned homotopy path generator, combining RL and imitation learning.

Problem

Learning a generalizable, robust neural controller that makes a dexterous robot hand track human hand-object kinematic references for diverse objects and manipulation skills, where RL needs task-specific rewards and trajectory optimization needs accurate dynamics.

Approach

Iteratively alternate between mining high-quality per-trajectory tracking demonstrations (RL with tracking-controller priors and homotopy optimization over simplified-to-complex reference paths, with a diffusion-based homotopy path generator) and training the neural controller jointly with RL reward and action-imitation loss on those demonstrations (data flywheel).

Key results

• On GRAB, the full method reaches 46.70% / 65.48% success vs 34.52% / 52.79% for DGrasp (best baseline reported around 38.58/54.82 for PPO w/o sup.), an improvement of over 10 points.
• On TACO, the full method reaches 48.77% / 74.38% success vs 38.42% / 47.78% for DGrasp and 34.98% / 57.64% for PPO w/o sup.
• Real-world tests on LEAP hand with Franka arm show the method succeeds on more objects than the PPO baseline (e.g., 75% vs 50% on several objects at the strictest level).

Measured effects reported in the paper

Surprising findings

• Training on tracking demonstrations for all GRAB trajectories with 16 GPUs for a week gave lower success (42.13%/60.41%) than the selective curated demonstration approach, suggesting data quality matters as much as quantity.

Limitations

• Acquiring high-quality tracking demonstrations is time-consuming
• Object can drop from the hand during in-hand manipulation; failures on new thin/novel categories
• Homotopy path generator generalizes poorly to out-of-distribution objects

Experimental setup

Datasets: GRAB TACO

Benchmarks: GRAB TACO

Metrics: object rotation error object translation error wrist error finger error success rate (10cm-20deg-0.8 and 10cm-40deg-1.2 thresholds)

Baselines: DGrasp PPO (OmniGrasp rew.) PPO (w/o sup., tracking rew.)

Builds on: PPO OmniGrasp DART DGrasp chain-of-thought

Closest papers by meaning

• ManipTrans: Efficient Dexterous Bimanual Manipulation Transfer via Residual Learning (CVPR 2025 · similarity 0.89)
• DexHandDiff: Interaction-aware Diffusion Planning for Adaptive Dexterous Manipulation (CVPR 2025 · similarity 0.89)
• DexFlyWheel: A Scalable and Self-improving Data Generation Framework for Dexterous Manipulation (NeurIPS 2025 · similarity 0.88)
• Scaffolding Dexterous Manipulation with Vision-Language Models (NeurIPS 2025 · similarity 0.88)
• Cross-Embodiment Dexterous Grasping with Reinforcement Learning (ICLR 2025 · similarity 0.87)
• SUGAR: A Scalable Human-Video-Driven Generalizable Humanoid Loco-Manipulation Learning Framework (NeurIPS 2026 · similarity 0.86)