Research map / 3D vision / 3D generation
VidBot: Learning Generalizable 3D Actions from In-the-Wild 2D Human Videos for Zero-Shot Robotic Manipulation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Extracts metric-scale 3D hand trajectories from in-the-wild monocular human videos (SfM + metric depth) and trains a coarse-to-fine diffusion affordance model with test-time cost guidance for zero-shot robot manipulation.
Problem
Learning embodiment-agnostic 3D affordances (contact points and interaction trajectories) from raw RGB-only in-the-wild human videos so robots can perform household manipulation zero-shot without teleoperated data.
Approach
Reconstructs temporally consistent, metric-scale 3D hand trajectories from monocular human videos using a depth foundation model plus SfM with gradient-based optimization, and factorizes the affordance model into a coarse pixel-space goal/contact predictor and a diffusion trajectory generator steered at test time by differentiable cost functions (multi-goal reaching, collision avoidance, contact n
Key results
- Achieves 88.2% average success rate across 13 simulated manipulation tasks, roughly 20% above the runner-up (Octo, 69.2%).
- Ablation: removing coarse goal prediction drops success from 85.6% to 57.8% on six tasks.
- Real-robot experiments on Hello Robot Stretch 3 and Boston Dynamics Spot achieve 80.0% success over 55 trials.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use VidBot instead of the strongest reported baseline, Octo* | Zero-shot simulator manipulation · 13 simulator manipulation tasks | success rate (%): 69.2 -> 88.2 | Helped |
| Add coarse goal prediction, goal conditioning, and multi-goal guidance (ablation) | VidBot · 6 simulator ablation tasks (Avg.) | success rate (%): 57.8 -> 85.6 | Helped |
| Add multi-goal guidance (ablation) | VidBot · 6 simulator ablation tasks (Avg.) | success rate (%): 73.3 -> 85.6 | Helped |
| Add contact-normal guidance (ablation) | VidBot · 6 simulator ablation tasks (Avg.) | success rate (%): 76.7 -> 85.6 | Helped |
| Add collision-avoidance guidance (ablation) | VidBot · 6 simulator ablation tasks (Avg.) | success rate (%): 77.8 -> 85.6 | Helped |
| Use final guidance cost as the trajectory-selection heuristic instead of random selection (ablation) | VidBot · 6 simulator ablation tasks (Avg.) | success rate (%): 74.5 -> 85.6 | Helped |
Limitations
- Data quality is bounded by depth foundation model and SfM accuracy despite loss-based filtering.
- Not suited to highly precise tasks such as unscrewing caps.
- Baselines VRB and GFlow were adapted to 3D or given standardized contacts, so comparison relies on modified protocols.
Experimental setup
Datasets: EpicKitchens-100 EpicFields SfM results FrankaKitchen PartManip ManiSkill
Benchmarks: FrankaKitchen PartManip ManiSkill IsaacGym real-robot tests
Metrics: success rate coincidental success
Baselines: GAPartNet Where2Act Octo VRB GFlow
Builds on: Diffuser VRB Perceiver Octo CLIP
Closest papers by meaning
- Robots Pre-train Robots: Manipulation-Centric Robotic Representation from Large-Scale Robot Datasets (ICLR 2025 · similarity 0.89)
- AffordDP: Generalizable Diffusion Policy with Transferable Affordance (CVPR 2025 · similarity 0.89)
- VidMan: Exploiting Implicit Dynamics from Video Diffusion Model for Effective Robot Manipulation (NeurIPS 2024 · similarity 0.88)
- Mitigating the Human-Robot Domain Discrepancy in Visual Pre-training for Robotic Manipulation (CVPR 2025 · similarity 0.88)
- IAAO: Interactive Affordance Learning for Articulated Objects in 3D Environments (CVPR 2025 · similarity 0.88)
- DiT4DiT: Jointly Modeling Video Dynamics and Actions for Generalizable Robot Control (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.