Research map / Video / Video-language understanding
ROVER: Recursive Reasoning Over Videos with Vision-Language Models for Embodied Tasks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ROVER, a recursive in-context framework that decomposes long robot videos into subtasks with a sliding frame window, improving VLM progress estimation, frame-level reasoning, and video QA while reducing hallucination.
Problem
VLMs struggle to reason frame-by-frame over long video sequences in embodied settings; reasoning over many frames causes hallucination (e.g., monotonic progress) and quadratic cost, while local reasoning loses global context.
Approach
A recursive decomposition where the VLM spawns child reasoning processes for subtasks, each using a sliding context window of three frames (first frame of the subtask, most recent frame, and the new frame), so context stays compact and time complexity scales linearly with video length.
Key results
- ROVER with Gemini-1.5-Pro achieves higher Pearson correlation with ground-truth progress than GVL, TemporalConcat, and LocalConcat, especially on non-expert trajectories.
- On 100 real-world and 100 simulation videos, total error rate drops from 76% (GVL) to 26% (ROVER) on real-world videos and from 57% to 23% on simulation videos.
- Hand-review of LLM judge showed 97.5% agreement with human judgement on 200 examples.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ROVER's recursive decomposition and sliding context window instead of GVL | Gemini-1.5-Pro video reasoning · Real-world videos (100 reviewed) | total error rate: 76% -> 26% | Helped |
| Use ROVER's recursive decomposition and sliding context window instead of GVL | Gemini-1.5-Pro video reasoning · Simulation videos (100 reviewed) | total error rate: 57% -> 23% | Helped |
| Use ROVER's recursive decomposition and sliding context window instead of GVL | Gemini-1.5-Pro video reasoning · Real-world videos (100 reviewed) | perception error rate: 47% -> 7% | Helped |
| Use ROVER's recursive decomposition and sliding context window instead of GVL | Gemini-1.5-Pro video reasoning · Simulation videos (100 reviewed) | perception error rate: 32% -> 5% | Helped |
| Add a sliding context window without recursive reasoning (ablation) | ROVER · Short-horizon tasks, including turning appliances on/off | correlation with ground-truth progress | Helped |
| Add recursive decomposition to the window-only variant (ablation) | ROVER · Tasks with numerous subtasks, such as pick-and-place | correlation with ground-truth progress | Helped |
Surprising findings
- GVL's and baselines' perception errors concentrate when more than 10 frames are in context (83% of GVL perception errors).
- The sliding window alone, without recursion, already reduces hallucination and matches or exceeds GVL on short-horizon tasks.
Limitations
- Decomposition errors (incorrect or redundant subtasks) can fragment reasoning.
- Implemented only with in-context learning; fine-tuning not evaluated.
- Relies on known object-centric subtask sequences per task.
Experimental setup
Datasets: RoboCasa expert demonstrations OpenX Embodiment videos
Benchmarks: frame-level task progress estimation (RoboCasa-based) frame-level natural language reasoning (RoboCasa-based) video QA (RoboCasa-based) OXE real-world progress estimation
Metrics: Pearson correlation L2 distance reasoning error rate reasoning success rate video QA accuracy precision recall temporal distance
Baselines: GVL LIV TemporalConcat LocalConcat
Builds on: Generative Value Learning (GVL) MimicGen RoboCasa
Closest papers by meaning
- Look, Remember and Reason: Grounded Reasoning in Videos with Language Models (ICLR 2024 · similarity 0.89)
- MVVBench: Benchmarking 4D Reasoning in Vision-Language Models (NeurIPS 2026 · similarity 0.89)
- Seeing Across Views: Benchmarking Spatial Reasoning of Vision-Language Models in Robotic Scenes (ICLR 2026 · similarity 0.88)
- Black Swan: Abductive and Defeasible Video Reasoning in Unpredictable Events (CVPR 2025 · similarity 0.88)
- PIVOT: Iterative Visual Prompting Elicits Actionable Knowledge for VLMs (ICML 2024 · similarity 0.88)
- Robot-R1: Reinforcement Learning for Enhanced Embodied Reasoning in Robotics (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.