Research map / Video / Video-language understanding
Modularized Self-Reflected Video Reasoner for Multimodal LLM with Application to Video Question Answering
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MSR-ViR, which combines a tree-structured modular spatial-temporal grounding module with a Multimodal LLM answerer and alternately trains an LLM question parser (via DPO using the answerer's loss as feedback) and the Multimodal LLM (via SFT) for interpretable VideoQA.
Problem
Video question answering with Multimodal LLMs is a black box: it cannot present reasoning paths or indicate which video segments and regions support the answer.
Approach
Integrates a modular spatial-temporal grounding network, driven by an LLM-generated JSON execution policy, into a Multimodal LLM, and trains the policy parser with DPO using the downstream answerer's loss as the preference signal, alternating with SFT of the answerer.
Key results
- MSR-ViR-L (LLaVA-NeXT based) reaches 74.9 average on NExT-QA vs 73.1 for LLaVA-NeXT and 71.9 for Qwen-VL, and 71.0 on STAR-sub vs 69.9 for LLaVA-NeXT.
- On zero-shot VideoMME, MSR-ViR-LV reaches 61.9 average vs 58.5 for LLaVA-Video; MSR-ViR-Q2 reaches 57.9 vs 55.3 for Qwen2-VL.
- On NExT-GQA, MSR-ViR-L obtains mIoU 23.4 and MSR-ViR-Q obtains Acc@GQA 18.5, exceeding grounding baselines such as SeViLa (mIoU 21.7).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add MSR-ViR modular grounding and self-reflection training | LLaVA-NeXT · NExT-QA | Average accuracy: 73.1 -> 74.9 | Helped |
| Add MSR-ViR modular grounding and self-reflection training | LLaVA-NeXT · STAR-sub | Average accuracy: 69.9 -> 71.0 | Helped |
| Add MSR-ViR modular grounding and self-reflection training | LLaVA-NeXT · EgoSchema subset | Accuracy: 54.8 -> 61.2 | Helped |
| Add MSR-ViR modular grounding and self-reflection training | LLaVA-NeXT · EgoSchema full | Accuracy: 43.3 -> 46.0 | Helped |
| Add MSR-ViR modular grounding and self-reflection training | LLaVA-Video · VideoMME | Average accuracy: 58.5 -> 61.9 | Helped |
| Add alternate self-reflection training (ablation) | MSR-ViRQ · NExT-QA | Average accuracy: 72.1 -> 73.6 | Helped |
| Add spatial localizer modules (ablation) | MSR-ViRQ · NExT-QA | Average accuracy: 72.2 -> 73.6 | Helped |
| Add instruction prompts (ablation) | MSR-ViRQ · NExT-QA | Average accuracy: 72.8 -> 73.6 | Helped |
| Add global video representation (ablation) | MSR-ViRQ · NExT-QA | Average accuracy: 70.4 -> 73.6 | Helped |
| Use only global video representation instead of the full visual input design (ablation) | MSR-ViRQ · NExT-QA | Average accuracy: 73.6 -> 68.3 | Hurt |
| Add reasoning paths (ablation) | MSR-ViRQ · NExT-GQA | Acc@QA: 66.8 -> 69.9 | Helped |
| Add reasoning paths (ablation) | MSR-ViRQ · NExT-GQA | Acc@GQA: 14.4 -> 18.5 | Helped |
Surprising findings
- On STAR-sub, MSR-ViR based on Qwen-VL (66.4) does not beat the LLaVA-NeXT direct baseline (69.9), and the gap to the best Multimodal LLM baseline is mixed across models.
Limitations
- Relies on a fixed set of seven small grounding modules (UniVTG, YOLO-World, and others), so grounding quality is bounded by these tools.
- Added computational overhead from running an LLM question parser and multiple grounding modules.
- Gains on STAR-sub and some benchmarks are mixed; the zero-shot long-video results rely on the base models' SFT.
Experimental setup
Datasets: NExT-QA STAR LLaVA-Video-178K NExT-GQA EgoSchema VideoMME
Benchmarks: NExT-QA STAR-sub EgoSchema VideoMME NExT-GQA
Metrics: accuracy mIoU IoU@0.5 Acc@GQA
Baselines: ATP MIST CoVGT HiTeA InternVideo BLIP-2 InstructBLIP SeViLa GCG TGB LLoVi MoReVQA
Builds on: Direct Preference Optimization UniVTG YOLO-World Qwen-VL LLaVA-NeXT
Closest papers by meaning
- LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understanding (NeurIPS 2024 · similarity 0.91)
- MMR-V: What's Left Unsaid? A Benchmark for Multimodal Deep Reasoning in Videos (ICLR 2026 · similarity 0.91)
- MoReVQA: Exploring Modular Reasoning Models for Video Question Answering (CVPR 2024 · similarity 0.91)
- Divide and Conquer: Exploring Language-centric Tree Reasoning for Video Question-Answering (ICML 2025 · similarity 0.90)
- VideoEspresso: A Large-Scale Chain-of-Thought Dataset for Fine-Grained Video Reasoning via Core Frame Selection (CVPR 2025 · similarity 0.89)
- Enhancing Video-LLM Reasoning via Agent-of-Thoughts Distillation (CVPR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.