Research map / Vision-language and multimodal / Multimodal LLMs
When Thinking Drifts: Evidential Grounding for Robust Video Reasoning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows chain-of-thought often hurts video QA in open MLLMs ("visual thinking drift") and proposes Visual Evidence Reward (VER), a GRPO reward that gives a bonus for correct reasoning traces citing question-relevant visual evidence, yielding Video-VER on Qwen2.5-VL-7B.
Problem
Chain-of-thought reasoning often degrades video understanding in MLLMs, producing verbose traces that hallucinate visual details or override correct direct answers.
Approach
Adds a binary visual evidence reward, judged by an auxiliary LLM against question-specific visual evidence generated offline by a stronger MLLM via inverted prompting (question plus ground-truth answer), into GRPO so reasoning is rewarded for grounding in the video.
Key results
- Video-VER (Qwen2.5-VL-7B, CoT) reaches 53.1 on VideoHallucer (+9.0 over base Qwen2.5-VL-7B with CoT) and 59.3 on Video-MME, ranking first on 9 of 10 benchmarks.
- Video-VER averages about +4.0 points over its base model across the 10 benchmarks.
- Qwen2.5-VL-7B with CoT underperforms direct answering on MVBench (59.8 vs 63.6) and Video-MME (54.7 vs 59.2), illustrating CoT degradation.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add Visual Evidence Reward during post-training | Qwen2.5-VL-7B CoT · MVBench | accuracy: 59.8 -> 64.1 | Helped |
| add Visual Evidence Reward during post-training | Qwen2.5-VL-7B CoT · VideoHallucer | accuracy: +9.0 | Helped |
| replace direct answering with CoT prompting | Qwen2.5-VL-7B · Video-MME | accuracy: 59.2 -> 54.7 | Hurt |
| replace direct answering with CoT prompting | Qwen2.5-VL-3B · MVBench | accuracy: 62.8 -> 57.4 | Hurt |
| replace direct answering with CoT prompting | Qwen2.5-VL-7B · EventHallusion | accuracy: 64.5 -> 67.3 | Helped |
| replace direct answering with CoT prompting | Qwen2.5-VL models · video reasoning tasks | accuracy | Mixed |
| use question-dependent visual evidence instead of general video captions (ablation) | Video-VER · MVBench | accuracy: 63.9 -> 64.1 | Helped |
| use question-dependent visual evidence instead of general video captions (ablation) | Video-VER · EventHallusion | accuracy: 70.3 -> 70.0 | Hurt |
| use question-dependent visual evidence instead of general video captions (ablation) | Video-VER · 10 video benchmarks | accuracy | Mixed |
| increase input frames from 16 to 32 (ablation) | Video-VER · Video-MME | accuracy: 56.0 -> 59.3 | Helped |
| increase input frames from 8 to 32 (ablation) | Video-VER · Vinoground Group | accuracy: 11.0 -> 14.4 | Helped |
| increase input frames across 8, 16, and 32 (ablation) | Video-VER · 10 video benchmarks | accuracy | Mixed |
Surprising findings
- CoT prompting lowers accuracy for open-source MLLMs such as Qwen2.5-VL on many video benchmarks compared with direct answering.
- Many erroneous video reasoning traces are logically sound but grounded in hallucinated visual details (visual thinking drift).
Limitations
- Focuses on moderate-length videos; long-video reasoning and sparse critical information remain open.
- Reward quality depends on the LLM judge (Llama-3.1-70B) and on the teacher's offline visual evidence, which may hallucinate.
- Evaluated only on closed-ended tasks such as multiple-choice QA; open-ended extension left for future work.
Experimental setup
Datasets: Video-R1-COT-165k Reversed-in-Time Video-R1-260k
Benchmarks: MVBench Video-MME VideoMMMU MMVU VideoHallucer EventHallusion VSI-Bench TempCompass TVBench Vinoground
Metrics: accuracy group score (Vinoground)
Baselines: GPT-4o LongVA Video-UTR LLaVA-OneVision Kangaroo TinyLLaVA-Video-R1 Video-R1 Qwen2.5-VL-3B Qwen2.5-VL-7B
Builds on: GRPO Video-R1 Qwen2.5-VL self-consistency
Closest papers by meaning
- See It, Say It, Sorted: An Iterative Training-Free Framework for Visually-Grounded Multimodal Reasoning in LVLMs (CVPR 2026 · similarity 0.91)
- Let VLMs Grade Their Own Thoughts: A Self-Quantification Approach to Reasoning-Aware Reward Modeling (CVPR 2026 · similarity 0.91)
- Video-of-Thought: Step-by-Step Video Reasoning from Perception to Cognition (ICML 2024 · similarity 0.91)
- VideoReasonBench: Can MLLMs Perform Vision-Centric Complex Video Reasoning? (ICLR 2026 · similarity 0.90)
- SLVR: Structured Latent Visual Reasoning via Human-like Reasoning Flows (NeurIPS 2026 · similarity 0.90)
- Act2See: Emergent Active Visual Perception for Video Reasoning (CVPR 2026 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.