Research map / Vision-language and multimodal / Visual question answering
MoReVQA: Exploring Modular Reasoning Models for Video Question Answering
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MoReVQA, a training-free three-stage modular (event parsing, grounding, reasoning) video QA system with external memory, plus a simple captioning baseline JCEF, achieving SOTA among training-free methods on NExT-QA, iVQA, EgoSchema and ActivityNet-QA.
Problem
Single-stage program-generation modular methods for video question answering are brittle because the planner must write full programs without grounding in video content, and they struggle with long, compositional videoQA.
Approach
Decompose planning into multi-stage LLM-generated API calls (event parsing, video grounding, then reasoning) that share an external memory, so each planning stage is informed by visual evidence from earlier stages; a simple caption-every-frame baseline (JCEF) already beats single-stage programs.
Key results
- MoReVQA reaches 69.2% on NExT-QA val, versus 64.0% for the upgraded single-stage ViperGPT+ and 66.7% for JCEF.
- On iVQA test, MoReVQA scores 60.9%, about 7 points above InstructBLIP (FlanT5XXL) at 53.8%.
- On EgoSchema test, MoReVQA scores 51.7%, versus 33.3% for the best prior method (LongViViT).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MoReVQA instead of JCEF | VideoQA with PALI-3 and PaLM-2 · NExT-QA | accuracy: 66.7 -> 69.2 | Helped |
| Use MoReVQA instead of JCEF | VideoQA with PALI-3 and PaLM-2 · iVQA | accuracy: 56.9 -> 60.9 | Helped |
| Use MoReVQA instead of JCEF | VideoQA with PALI-3 and PaLM-2 · EgoSchema | accuracy: 50.0 -> 51.7 | Helped |
| Use MoReVQA instead of JCEF | VideoQA with PALI-3 and PaLM-2 · ActivityNet-QA | accuracy: 43.3 -> 45.3 | Helped |
| Add event parsing and reasoning to the no-stage JCEF baseline (ablation) | MoReVQA · NExT-QA val | accuracy: 66.7 -> 68.3 | Helped |
| Add event parsing and reasoning to the no-stage JCEF baseline (ablation) | MoReVQA · iVQA test | accuracy: 56.9 -> 56.9 | No clear effect |
| Add event parsing and grounding to the no-stage JCEF baseline (ablation) | MoReVQA · NExT-QA val | accuracy: 66.7 -> 68.7 | Helped |
| Add event parsing and grounding to the no-stage JCEF baseline (ablation) | MoReVQA · iVQA test | accuracy: 56.9 -> 57.5 | Helped |
| Add grounding to event parsing and reasoning (ablation) | MoReVQA · NExT-QA val | accuracy: 68.3 -> 69.2 | Helped |
| Add grounding to event parsing and reasoning (ablation) | MoReVQA · iVQA test | accuracy: 56.9 -> 60.9 | Helped |
| Add reasoning to event parsing and grounding (ablation) | MoReVQA · NExT-QA val | accuracy: 68.7 -> 69.2 | Helped |
| Add reasoning to event parsing and grounding (ablation) | MoReVQA · iVQA test | accuracy: 57.5 -> 60.9 | Helped |
Surprising findings
- A simple caption-every-frame baseline (JCEF) outperforms the single-stage ViperGPT+ program-generation system on all four datasets.
- LLM-only prompting scores 41.0% on EgoSchema, suggesting language bias in that dataset.
Limitations
- Relies on off-the-shelf frozen modules and LLM prompting; performance bounded by those components.
- Single-stage ViperGPT+ reimplementation is an upgraded version, so comparisons depend on the authors' reimplementation.
- Frame sampling choices (e.g., n=16 or 30 frames) affect results and cost.
Experimental setup
Datasets: NExT-QA iVQA EgoSchema ActivityNet-QA NExT-GQA ActivityNet-Para
Benchmarks: NExT-QA iVQA EgoSchema ActivityNet-QA NExT-GQA ActivityNet-Para
Metrics: accuracy mIoP Acc@GQA CIDEr
Baselines: ViperGPT ViperGPT+ JCEF LLM-only SeViLA InstructBLIP BLIP-2 Vid2Seq MIST-CLIP
Builds on: ViperGPT Socratic models VisProg OWL-ViT CLIP
Closest papers by meaning
- Divide and Conquer: Exploring Language-centric Tree Reasoning for Video Question-Answering (ICML 2025 · similarity 0.91)
- Modularized Self-Reflected Video Reasoner for Multimodal LLM with Application to Video Question Answering (ICML 2025 · similarity 0.91)
- Commonsense Video Question Answering through Video-Grounded Entailment Tree Reasoning (CVPR 2025 · similarity 0.90)
- VideoEspresso: A Large-Scale Chain-of-Thought Dataset for Fine-Grained Video Reasoning via Core Frame Selection (CVPR 2025 · similarity 0.90)
- Enhancing Video-LLM Reasoning via Agent-of-Thoughts Distillation (CVPR 2025 · similarity 0.89)
- Tool-Augmented Spatiotemporal Reasoning for Streamlining Video Question Answering Task (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.