Research map / Video / Video-language understanding
Video Language Planning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes video language planning (VLP), a tree search that combines a fine-tuned PaLM-E VLM (as policy and heuristic value function) with a text-to-video diffusion model (as dynamics model) to synthesize long-horizon video plans, executed via goal-conditioned policies.
Problem
Long-horizon robotic planning requires reasoning about both high-level semantic task steps and low-level visual dynamics, which VLMs trained on static images and text-only LLMs handle poorly, while text-to-video models only generate short clips.
Approach
Forward tree search (parallel hill climbing beam search) alternates VLM-proposed text actions, text-to-video rollouts as dynamics, and VLM heuristic scoring of frames to build hundreds-of-frames video plans, then converts frames into actions with a goal-conditioned policy conditioned on each intermediate frame.
Key results
- On simulated Language Table long-horizon execution, VLP reaches 64% completion on move-to-area, 92% on group-by-color and 16% on make-line, outperforming UniPi, LAVA, RT-2 and PaLM-E baselines.
- In video plan synthesis, VLP achieves 58%, 98% and 66% on simulated move-area, group-color and make-line, versus 2%, 4% and 2% for UniPi.
- Increasing planning budget (beams and branching factor) steadily improves make-line execution from 0% to 16% completion.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use VLP instead of LAVA | long-horizon robot execution · Move to Area | Completion: 22% -> 64% | Helped |
| use VLP instead of RT-2 | long-horizon robot execution · Group by Color | Completion: 26% -> 92% | Helped |
| use VLP instead of UniPi | long-horizon robot execution · Make Line | Completion: 4% -> 16% | Helped |
| add the value function to VLP (ablation) | video plan generation · Move Area, simulated | Video plan success: 10% -> 58% | Helped |
| add the value function to VLP (ablation) | video plan generation · Group Color, simulated | Video plan success: 42 % -> 98% | Helped |
| add the value function to VLP (ablation) | video plan generation · Make Line, simulated | Video plan success: 8% -> 66% | Helped |
| increase search branching factor from 4 to 16 (ablation) | VLP · Make Line | Completion: 0% -> 2% | Helped |
| increase planning horizon from 1 to 2 (ablation) | VLP with 1 beam and branching factor 16 · Make Line | Completion: 2% -> 8% | Helped |
| increase the number of beams from 1 to 2 (ablation) | VLP with horizon 2 and branching factor 16 · Make Line | Completion: 8% -> 16% | Helped |
| apply the goal-conditioned policy to every intermediate frame instead of only the last frame (ablation) | action extraction from VLP video plans · Group Color | Completion: 66% -> 92% | Helped |
| replace inverse dynamics with a goal-conditioned policy applied to every intermediate frame | action extraction from VLP video plans · Group Color | Completion: 80% -> 92% | Helped |
Limitations
- Videos sometimes show objects spontaneously appearing, disappearing, or teleporting, indicating inaccurate dynamics.
- Images as the world state cannot capture 3D state, physics, or mass.
- Long planning is computationally expensive (about 30 minutes per video plan, about 1 hour per execution episode).
Experimental setup
Datasets: Language Table RT-1 Bridge RT-2 data Ego4D EPIC-KITCHENS LAION-400M
Benchmarks: Language Table (simulation) Language Table (real robot)
Metrics: task completion rate reward score
Baselines: UniPi LAVA RT-2 PaLM-E VLP (No Value Function)
Builds on: PaLM-E UniPi LAVA SayCan parallel hill climbing
Closest papers by meaning
- FLIP: Flow-Centric Generative Planning as General-Purpose Manipulation World Model (ICLR 2025 · similarity 0.90)
- VLA-OS: Structuring and Dissecting Planning Representations and Paradigms in Vision-Language-Action Models (NeurIPS 2025 · similarity 0.90)
- VideoVLA: Video Generators Can Be Generalizable Robot Manipulators (NeurIPS 2025 · similarity 0.90)
- VLMimic: Vision Language Models are Visual Imitation Learner for Fine-grained Actions (NeurIPS 2024 · similarity 0.89)
- Visual Planning: Let's Think Only with Images (ICLR 2026 · similarity 0.89)
- Plan-Seq-Learn: Language Model Guided RL for Solving Long Horizon Robotics Tasks (ICLR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.