Research map / Video / Action recognition
Grounding Video Models to Actions through Goal Conditioned Exploration
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Grounds a pretrained-style video model to continuous robot actions by self-exploration, using generated video frames as goals for a chunked diffusion-policy goal-conditioned controller trained via hindsight relabeling, with no action labels or rewards.
Problem
Video models generate plausible visual plans but do not specify how an embodied agent should actuate to reach the depicted states; existing grounding methods need action-labeled demonstrations or rigid-object flow estimates.
Approach
Use a video model's generated frames as visual goals for online goal-conditioned exploration, storing rollouts in a replay buffer and relabeling them in hindsight to train a goal-conditioned diffusion policy, augmented with chunked action generation and periodic random action bootstrapping.
Key results
- On 8 Libero tasks, the method reaches 62.0% overall success vs 19.4% for BC (with expert actions) and 0.3% for AVDC on the second scene, without action labels.
- On 6 Meta-World tasks, the method achieves 67.2% average success, above DP BC (51.7%) and AVDC (50.7%).
- On iTHOR navigation, the method gets 51.0% (Bedroom) and 52.7% (Bathroom) average, on par with BC-based baselines and far above AVDC (26.7%, 6.1%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use video-guided exploration instead of purely random exploration (ablation) | Goal-conditioned exploration policy · Libero tasks | task success | Helped |
| Add random-action bootstrapping before training and periodic random exploration during training (ablation) | Goal-conditioned exploration policy · Libero tasks | task success during exploration | Helped |
| Add extra periodic random exploration during training (ablation) | Goal-conditioned exploration policy · Libero Scene 1 | exploration task success | Helped |
| Use chunk-level action prediction instead of single-action prediction (ablation) | Goal-conditioned exploration policy · Libero | test-time success rate: +25.8% and 21.0% | Helped |
| Increase video-model training data from 10 to 50 demonstrations per task (ablation) | Video-guided exploration policy · Libero Scene 1 | success rate: 29.2 ± 5.2 -> 55.4 ± 11.1 | Helped |
| Increase video-model training data from 10 to 50 demonstrations per task (ablation) | Video-guided exploration policy · Libero Scene 2 | success rate: 57.2 ± 5.6 -> 67.8 ± 7.6 | Helped |
| Increase video-model horizon from 7 to 8 (ablation) | Goal-conditioned exploration policy · Libero Scene 1 | success rate: 42.2 ± 8.9 -> 48.6 ± 6.6 | Helped |
| Increase video-model horizon from 7 to 9 (ablation) | Goal-conditioned exploration policy · Libero Scene 1 | success rate: 42.2 ± 8.9 -> 38.0 ± 6.7 | Hurt |
| Increase video-model horizon from 7 to 8 (ablation) | Goal-conditioned exploration policy · Libero Scene 2 | success rate: 62.0 ± 8.7 -> 52.8 ± 7.6 | Hurt |
| Increase goal-conditioned policy horizon from 16 to 24 (ablation) | Goal-conditioned exploration policy · Libero Scene 2 | success rate: 62.0 ± 8.7 -> 46.8 ± 10.9 | Hurt |
| Increase video exploration frequency from q_v = 800 to the default q_v = 200 (ablation) | Goal-conditioned exploration policy · Libero | success rate | No clear effect |
| Train for more iterations, from 40k to 200k steps (ablation) | Goal-conditioned exploration policy · Libero Scene 1 | success rate: 33.2 ± 8.6 -> 42.2 ± 8.9 | Mixed |
Surprising findings
- Without random action bootstrapping the agent collapses and achieves essentially no test-time success.
- Pure random exploration without video guidance fails across all Libero tasks.
Limitations
- Requires dozens of video-guided rollouts per task, and resets are easy only in simulation.
- Random exploration in the physical world may cause undesired contacts; struggles with millimeter-precision tasks such as block stacking.
- Evaluated only in simulation; video models are trained from scratch per environment rather than large pretrained text-conditioned models.
Experimental setup
Datasets: LIBERO demonstrations Meta-World demonstrations (AVDC checkpoint) CALVIN iTHOR demonstrations
Benchmarks: LIBERO Meta-World CALVIN iTHOR
Metrics: success rate
Baselines: BC GCBC DP BC DP GCBC SuSIE AVDC DrQ+LIV
Builds on: AVDC Diffusion Policy SuSIE DrQ
Closest papers by meaning
- OSVI-WM: One-Shot Visual Imitation for Unseen Tasks using World-Model-Guided Trajectory Generation (NeurIPS 2025 · similarity 0.89)
- Learning to Act from Actionless Videos through Dense Correspondences (ICLR 2024 · similarity 0.88)
- DiT4DiT: Jointly Modeling Video Dynamics and Actions for Generalizable Robot Control (NeurIPS 2026 · similarity 0.88)
- World Models as Group Actions (NeurIPS 2026 · similarity 0.88)
- Learning 3D Persistent Embodied World Models (NeurIPS 2025 · similarity 0.88)
- Navigation World Models (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.