Research map / Reinforcement learning / Visual and representation RL
Scaffolding Dexterous Manipulation with Vision-Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Uses an off-the-shelf VLM (Gemini 2.5 Flash Thinking) to produce 2D/3D keypoint and wrist trajectory "scaffolds" from a language command and scene image, then trains a residual RL policy in simulation to track them for dexterous Allegro-hand manipulation, transferred zero-shot to a real robot.
Problem
Dexterous manipulation with RL requires hand-crafted task-specific rewards or demonstration-based reference trajectories, which are costly to collect and limit scalability and generalization.
Approach
Replace human demonstrations with coarse VLM-generated 3D keypoint and wrist trajectories ("scaffolds"); the low-level residual RL policy is rewarded by keypoint tracking and contact, requiring no demonstrations or handcrafted rewards, and VLM plans can be refined via in-context examples of successful plans.
Key results
- Few-shot VLM scaffold method averages 72% success across 8 simulated tasks, and 81% after iterative refinement, approaching the oracle scripted-plan performance.
- Real-world success rates of 90% (Place Bottle onto Plate), 85% (Slide Box to Bottle), and 65% (Hammer Three Times) over 20 rollouts each.
- Incomplete trajectory tracking accounts for 26% of rollouts, the dominant failure mode, while keypoint detection errors account for 5%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Provide three successful plans as in-context examples | VLM scaffolding · Simulation task suite | success rate | Helped |
| Iteratively provide successful training-set plans as in-context examples (ablation) | VLM scaffolding · Simulation task suite | success rate: 81% | Helped |
| Use fewer than 10 waypoints (ablation) | VLM scaffolding · Simulation tasks | success rate | Hurt |
| Use 20 rather than 10 waypoints (ablation) | VLM scaffolding · Hammer task | success rate | Helped |
| Replace VLM keypoints with hand-specified oracle keypoints (ablation) | VLM scaffolding · Sponge task | success rate | Helped |
| Replace VLM-generated trajectories with scripted oracle plans (ablation) | VLM scaffolding · Drawer task | success rate | Helped |
| Replace Gemini 2.5 Flash Thinking with Molmo | VLM scaffold generation · Drawer, Hammer, and Pliers | trajectory feasibility: 87 vs 0; 67 vs 0; 80 vs 0 | Hurt |
| Use Molmo instead of Gemini 2.5 Flash Thinking for keypoint detection | VLM scaffold generation · Drawer, Hammer, and Pliers | keypoint accuracy: 100 vs 100; 100 vs 87; 100 vs 90 | Mixed |
Surprising findings
- Open VLM Molmo identified correct keypoints in all tested cases but produced 0% feasible trajectories, while Gemini 2.5 Flash Thinking produced feasible trajectories in 67-87% of trials.
- Pre-recorded trajectory reuse performs poorly on most tasks, and perfect-demonstration diffusion policy results are only comparable to the method with 50 demonstrations.
Limitations
- Assumes rigid-body interactions, limiting applicability to deformable objects.
- VLM reasoning latency of 1-2 minutes restricts real-time responsiveness, and plans do not account for low-level controller capabilities.
- Sim-to-real gap remains challenging for high-precision tasks; keypoint tracking degrades under occlusion.
Experimental setup
Datasets: ReplicaCAD scenes RoboCasa objects
Benchmarks: ManiSkill3 8-task dexterous suite (Move Apple, Move Bottle, Open Drawer, Open Fridge, Hammer, Wipe with Sponge, Close Sc real-world tasks: Place Bottle onto Plate, Slide Box to Bottle, Hammer Three Times
Metrics: success rate standard error over 3 seeds
Baselines: Iterative Keypoint Rewards (IKER) Pre-recorded Trajectories Oracle Keypoints and Trajectories Reduced Waypoints RL with Simple Reward RL with Complex Reward Diffusion Policy
Builds on: Iterative Keypoint Rewards residual reinforcement learning PPO ManiSkill
Closest papers by meaning
- Vision-Language-Action Pretraining from Large-Scale Human Videos (ICML 2026 · similarity 0.92)
- Cross-Embodiment Dexterous Grasping with Reinforcement Learning (ICLR 2025 · similarity 0.91)
- ManipTrans: Efficient Dexterous Bimanual Manipulation Transfer via Residual Learning (CVPR 2025 · similarity 0.91)
- Object-Centric Prompt-Driven Vision-Language-Action Model for Robotic Manipulation (CVPR 2025 · similarity 0.89)
- SparseDFF: Sparse-View Feature Distillation for One-Shot Dexterous Manipulation (ICLR 2024 · similarity 0.89)
- Grasp2Grasp: Vision-Based Dexterous Grasp Translation via Schrödinger Bridges (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.