Research map / Video / Video-language understanding
Searching Videos as Trees: Self-Correcting Agents for Grounded Long Video QA
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VideoTreeSearch, an agent that navigates an adaptive CLIP-based scene-boundary tree with zoom_in/zoom_out/shift/answer actions, trained via detour-and-recovery trajectory SFT followed by GRPO, reaching SOTA on grounded long-video QA benchmarks.
Problem
Grounded long-video QA requires answering a question while localizing the supporting short interval; existing multi-turn cropping agents use a single continuous crop action, converge prematurely, and cannot backtrack from early mistakes.
Approach
Organize the video as a non-uniform temporal tree built from CLIP scene boundaries and let the agent navigate it with discrete descend, ascend, lateral-move and answer actions, training it on synthesized trajectories that include deliberate wrong-branch detours and recovery.
Key results
- VTS reaches 16.8 mIoU and 36.4 accuracy on CG-Bench, +12.5 mIoU over the strongest cropping baseline LongVT.
- VTS reaches 18.4 T-F1 on Haystack-Ego4D, +7.4 over TimeSearch-R.
- VTS achieves 67.5 on Video-MME, 58.2 on MLVU, and 54.7 on LVBench, outperforming multi-turn agentic baselines.
Surprising findings
- Training on optimal-path-only trajectories underperforms detour-and-recovery trajectories by 6.4 mIoU on CG-Bench.
- Flat tree slightly beats hierarchical tree on Haystack-Ego4D grounding (18.9 vs 18.4 T-F1).
Limitations
- Produces a single evidence interval per question and cannot handle evidence spread across disjoint temporal regions.
- CLIP-based scene boundary detection is less reliable on visually homogeneous video.
- Trajectory synthesis depends on external VLM and LLM models, bounding training signal quality.
Experimental setup
Datasets: CG-Bench Haystack-Ego4D LongClueQA LongVideoBench
Benchmarks: CG-Bench Haystack-LVBench Haystack-Ego4D Video-MME MLVU LVBench
Metrics: mIoU T-F1 accuracy
Baselines: Qwen3-VL-8B Video-R1 SiLVR LongVT Video-o3 VideoZoomer TimeSearch-R
Builds on: GRPO VideoTree VideoMiner CLIP
Closest papers by meaning
- Grounded Question-Answering in Long Egocentric Videos (CVPR 2024 · similarity 0.90)
- MAGNET: A Multi-agent Framework for Finding Audio-Visual Needles by Reasoning over Multi-Video Haystacks (NeurIPS 2025 · similarity 0.89)
- Grounded Multi-Hop VideoQA in Long-Form Egocentric Videos (AAAI 2025 · similarity 0.89)
- LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understanding (NeurIPS 2024 · similarity 0.88)
- Commonsense Video Question Answering through Video-Grounded Entailment Tree Reasoning (CVPR 2025 · similarity 0.88)
- BIMBA: Selective-Scan Compression for Long-Range Video Question Answering (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.