Research map / Video / Video-language understanding
Tool-Augmented Spatiotemporal Reasoning for Streamlining Video Question Answering Task
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Equips GPT-4o with a 22-tool Video Toolkit and a training-free STAR planner that alternates temporal and spatial tools to progressively localize 3D regions of interest for VideoQA.
Problem
MLLMs struggle to jointly model spatial relationships within video frames and causal temporal dynamics on reasoning-intensive VideoQA; tool-augmented approaches use a single dimension, unbalanced toolsets, and suffer from toolchain shortcut behavior with weak scheduling.
Approach
A training-free planner (GPT-4o) that uses a visible frame dictionary and enforces alternating temporal and spatial tool calls, with general tools only as a last resort, to progressively narrow a 3D region of interest and avoid toolchain shortcuts.
Key results
- STAR with GPT-4o reaches 70.0% on VideoMME (+8.2 over GPT-4o at 32 frames) using about 30 frames and 15.8 s runtime.
- STAR achieves 57.2% on LongVideoBench validation (+4.6 over GPT-4o at 32 frames), above Qwen2.5-VL-7B (53.7%) and InternVL3-8B (48.9%).
- On NExT-QA test, STAR reaches 82.1% with 7.2 frames, versus 80.4% for the best prior method (Detector-based T*).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| augment GPT-4o with STAR and the Video Toolkit | GPT-4o · VideoMME | accuracy: 61.8 -> 70.0 | Helped |
| augment GPT-4o with STAR and the Video Toolkit | GPT-4o · LongVideoBench Validation Set | accuracy: 52.6 -> 57.2 | Helped |
| use STAR's spatiotemporal-interleaved toolchain instead of temporal-then-spatial disentanglement (ablation) | STAR · VideoMME | accuracy: 68.6 -> 70.0 | Helped |
| remove tool invocation constraints (ablation) | STAR · VideoMME | accuracy: 70.0 -> 61.2 | Hurt |
| replace explicit toolchain constraints with prompting for longer chains (ablation) | STAR · VideoMME | accuracy: 70.0 -> 60.4 | Hurt |
| provide annotated long-toolchain examples for in-context learning (ablation) | STAR · VideoMME | accuracy: 70.0 -> 63.2 | Hurt |
| remove the Frame Selector (ablation) | STAR · VideoMME | accuracy: Acc Drop 4.6 | Hurt |
| remove the Temporal Grounding Tool (ablation) | STAR · VideoMME | accuracy: Acc Drop 0.9 | Hurt |
| remove the Object Detector (ablation) | STAR · VideoMME | accuracy: Acc Drop 1.4 | Hurt |
| remove the Image QA Tool (ablation) | STAR · VideoMME | accuracy: Acc Drop 1.5 | Hurt |
| remove the Video QA Tool (ablation) | STAR · VideoMME | accuracy: Acc Drop 2.3 | Hurt |
| remove the Multiple Image QA Tool (ablation) | STAR · VideoMME | accuracy: Acc Drop 0.9 | Hurt |
Surprising findings
- Plain no-constraint tool planning often collapses into toolchain shortcuts where the planner calls the video QA tool directly, using 33% of calls on general tools.
- Prompting alone for longer toolchains yields only marginal gains (61.2 to 60.4 accuracy in one setting), suggesting prompt-based control is insufficient.
Limitations
- Relies on GPT-4o planner APIs, causing API costs from repeated invocations.
- Does not use subtitles or audio, which hurts performance on some questions (e.g., LongVideoBench and failure cases).
- Sparse frame sampling misses global video themes; struggles with inferring motivations behind actions.
Experimental setup
Datasets: VideoMME NExT-QA LongVideoBench EgoSchema
Benchmarks: VideoMME NExT-QA LongVideoBench
Metrics: accuracy number of processed frames runtime toolchain length number of tools
Baselines: GPT-4o GPT-4v Gemini 1.5 Pro Claude 3.5 Sonnet Qwen2-VL Qwen2.5-VL InternVL2.5 InternVL3 VideoLLaMA3 mPLUG-Owl3 VideoAgent VideoTree
Builds on: ReAct OctoTools Set-of-Mark prompting T* AKeyS
Closest papers by meaning
- VideoEspresso: A Large-Scale Chain-of-Thought Dataset for Fine-Grained Video Reasoning via Core Frame Selection (CVPR 2025 · similarity 0.91)
- STAIR: Spatial-Temporal Reasoning with Auditable Intermediate Results for Video Question Answering (AAAI 2024 · similarity 0.89)
- Divide and Conquer: Exploring Language-centric Tree Reasoning for Video Question-Answering (ICML 2025 · similarity 0.89)
- Language-aware Visual Semantic Distillation for Video Question Answering (CVPR 2024 · similarity 0.89)
- STEP: Enhancing Video-LLMs' Compositional Reasoning by Spatio-Temporal Graph-guided Self-Training (CVPR 2025 · similarity 0.89)
- Modularized Self-Reflected Video Reasoner for Multimodal LLM with Application to Video Question Answering (ICML 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.