Research map / Vision-language and multimodal / Multimodal LLMs
VideoChat-R1.5: Visual Test-Time Scaling to Reinforce Multimodal Reasoning by Iterative Perception
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Visual Test-Time Scaling (VTTS), which iteratively re-perceives predicted spatio-temporal regions during inference, trained with GRPO-style RL on a new VTTS-80K dataset to yield VideoChat-R1.5 models on Qwen2.5-VL-3B/7B.
Problem
MLLMs typically perceive visual inputs only once before reasoning, so their reasoning is limited by a static perception stage and cannot recover from poor initial evidence; spatio-temporal grounding is also hard to learn from standard text-only autoregressive supervision.
Approach
An Iterative Perception mechanism where the model predicts spatio-temporal ROIs from its current reasoning, re-samples those regions (dense frames or cropped images) at higher detail, and repeats; ROI prediction is learned via RL with IoU-based clue rewards alongside answer and format rewards.
Key results
- VideoChat-R1.5-7B with multi-iteration inference reaches 67.1 on VideoMME and 62.6 on LongVideoBench versus 64.4 and 56.0 for Qwen2.5-VL-7B.
- On NextGQA, VideoChat-R1.5-7B reaches Acc@GQA 61.9 and mIoP 74.9 versus 42.3 and 54.0 for Qwen2.5-VL-7B.
- On Charades-STA after fine-tuning, VideoChat-R1.5-7B reaches mIoU 60.6 and R@0.5 71.6 versus 43.6 and 42.9 for Qwen2.5-VL-7B.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply three-step iterative perception instead of single-step perception (ablation) | VideoChat-R1.5-7B · LongVideoBench | accuracy: 61.4 -> 62.6 | Helped |
| Apply three-step iterative perception instead of single-step perception (ablation) | VideoChat-R1.5-7B · VideoMME | accuracy: 65.2 -> 67.1 | Helped |
| Use reinforcement fine-tuning instead of SFT (ablation) | Qwen2.5-VL-7B · Charades-STA | mIoU: 54.9 -> 60.6 | Helped |
| Use reinforcement fine-tuning instead of SFT (ablation) | Qwen2.5-VL-7B · OCRBench | Score: 815 -> 865 | Helped |
| Increase perception iterations (ablation) | VideoChat-R1.5 · VideoMME | score: 65.2 to 67.9 | Helped |
| Increase perception iterations (ablation) | VideoChat-R1.5 · LongVideoBench | score: 61.4 to 62.9 | Helped |
| Increase perception iterations (ablation) | VideoChat-R1.5 · OCRBench | score: 856 to 866 | Helped |
| Apply VTTS with reinforcement fine-tuning | Qwen2.5-VL-7B · VideoMME | accuracy: 64.4 -> 67.1 | Helped |
| Apply VTTS with reinforcement fine-tuning | Qwen2.5-VL-7B · NextGQA | Acc@GQA: 42.3 -> 61.9 | Helped |
| Apply VTTS with reinforcement fine-tuning | Qwen2.5-VL-7B · ReXTime | Acc: 70.4 -> 74.8 | Helped |
| Apply VTTS with reinforcement fine-tuning | Qwen2.5-VL-7B · GoT | AO: 12.6 -> 52.2 | Helped |
Surprising findings
- SFT on the same VTTS-80K data degrades the backbone (e.g., VideoMME 63.9 vs 65.1 base and OCRBench 815 vs 856), while RL improves it.
- Under the same training data, VideoMME is only modestly improved by RL without the QA data (64.8 to 65.9 range), suggesting gains depend on dataset composition.
Limitations
- Iterative perception increases inference time, which may hinder real-time use.
- Validated only in the visual domain; the inference process is not optimized with caching or visual-language priors.
- Failures stem from inaccurate localization of critical clues and loss of global context.
Experimental setup
Datasets: VTTS-80K LLaVA-Video PerceptionTest STAR Charades-STA NextGQA Multimodal-Open-R1-8k Geo ScienceQA GOT-10k RefCOCO VisualCoT
Benchmarks: MVBench PerceptionTest LongVideoBench VideoMMMU MLVU VideoMME LVBench NextGQA ReXTime Charades-STA QVHighlight ActivityNet
Metrics: accuracy mIoU IoU@0.5 IoP Acc@GQA recall@IoU success rate average overlap
Baselines: Qwen2.5-VL-3B Qwen2.5-VL-7B VideoChat-TPO VideoChat-Flash InternVL2.5 GPT-4o Gemini-1.5-Pro VideoChat-R1 LLaVA-CoT
Builds on: GRPO Qwen2.5-VL DeepSeek-R1 Visual CoT
Closest papers by meaning
- Perception-R1: Advancing Multimodal Reasoning Capabilities of MLLMs via Visual Perception Reward (ICLR 2026 · similarity 0.91)
- Unleashing Perception-Time Scaling to Multimodal Reasoning Models (ICLR 2026 · similarity 0.91)
- Incentivizing Versatile Video Reasoning in MLLMs via Data-Efficient Reinforcement Learning (CVPR 2026 · similarity 0.90)
- SPARC: Separating Perception And Reasoning Circuits for Test-time Scaling of VLMs (ICML 2026 · similarity 0.90)
- See It, Say It, Sorted: An Iterative Training-Free Framework for Visually-Grounded Multimodal Reasoning in LVLMs (CVPR 2026 · similarity 0.90)
- GThinker: Towards General Multimodal Reasoning via Cue-Guided Rethinking (CVPR 2026 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.