Research map / Vision-language and multimodal / Multimodal LLMs
VLM-R³: Region Recognition, Reasoning, and Refinement for Enhanced Multimodal Chain-of-Thought
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VLM-R³, a Qwen2.5-VL-7B-based MLLM that interleaves region cropping with textual chain-of-thought, trained with SFT on a curated VLIR dataset and then R-GRPO reinforcement learning.
Problem
Current reasoning MLLMs keep reasoning mostly in text and only ground visual evidence once, struggling with tasks needing dynamic, iterative revisiting of image regions.
Approach
An interactive crop tool inside the reasoning loop (model emits bbox JSON, the cropped and zoomed region is re-injected as visual tokens), trained with a GRPO variant (R-GRPO) that masks policy gradients on injected image tokens and adds region-validity and reasoning-length rewards.
Key results
- On MathVista, VLM-R³ reaches 70.4 vs 68.2 for the Qwen2.5-VL 7B base; on MathVision 30.2 vs 25.1.
- On ScienceQA, VLM-R³ reaches 87.9 vs 73.6 for the base model, the largest reported gain.
- Ablation: removing R-GRPO drops average from 79.3 to 77.8; removing VLIR fine-tuning drops it to 72.1.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| remove interleaved region images while retaining bounding-box localization (ablation) | VLM-R³ · MathVista | score: 67.1 -> 70.4 | Helped |
| remove interleaved region images while retaining bounding-box localization (ablation) | VLM-R³ · MMMU | score: 59.4 -> 62.2 | Helped |
| remove interleaved region images while retaining bounding-box localization (ablation) | VLM-R³ · ScienceQA | score: 75.4 -> 87.9 | Helped |
| remove interleaved region images while retaining bounding-box localization (ablation) | VLM-R³ · DocVQA | score: 95.9 -> 96.8 | Helped |
| skip VLIR fine-tuning before R-GRPO (ablation) | VLM-R³ · MathVista | score: 65.8 -> 70.4 | Helped |
| skip VLIR fine-tuning before R-GRPO (ablation) | VLM-R³ · MMMU | score: 57.0 -> 62.2 | Helped |
| skip VLIR fine-tuning before R-GRPO (ablation) | VLM-R³ · ScienceQA | score: 72.2 -> 87.9 | Helped |
| skip VLIR fine-tuning before R-GRPO (ablation) | VLM-R³ · DocVQA | score: 93.3 -> 96.8 | Helped |
| remove R-GRPO after VLIR supervised fine-tuning (ablation) | VLM-R³ · MathVista | score: 69.7 -> 70.4 | Helped |
| remove R-GRPO after VLIR supervised fine-tuning (ablation) | VLM-R³ · MMMU | score: 60.8 -> 62.2 | Helped |
| remove R-GRPO after VLIR supervised fine-tuning (ablation) | VLM-R³ · ScienceQA | score: 84.6 -> 87.9 | Helped |
| remove R-GRPO after VLIR supervised fine-tuning (ablation) | VLM-R³ · DocVQA | score: 96.1 -> 96.8 | Helped |
Surprising findings
- Removing interleaved chain-of-thought improves MMMU and ScienceQA-related ablation numbers relative to some variants, and the reported ScienceQA ablation numbers are internally inconsistent (w/o interleaved CoT scores 75.4 yet the drop is stated as 12.5).
Limitations
- VLIR dataset is small (11,810 samples) and built largely from GPT-4o and Qwen2.5-VL 72B outputs; no release stated.
- Reported numbers have internal inconsistencies (ablation drops vs. table values, ScienceQA base score 73.6 vs 73.57), and ablation/grounding analyses are limited to few benchmarks.
- Training uses only about 5,000 RL samples and frozen vision encoder and projector.
Experimental setup
Datasets: VLIR GQA TextVQA DocVQA InfographicsVQA VSR M3CoT
Benchmarks: MathVista MathVision MMMU MME ScienceQA DocVQA HallusionBench
Metrics: accuracy
Baselines: Qwen2.5-VL 7B InternVL2.5-8B LLaVA-Next 8B Gemini-2 Flash GPT-4o LLaVA-CoT Mulberry R1-onevision Vision-R1 o1
Builds on: GRPO Visual CoT Qwen2.5-VL DeepSeek R1
Closest papers by meaning
- Multi-step Visual Reasoning with Visual Tokens Scaling and Verification (NeurIPS 2025 · similarity 0.93)
- VGR: Visual Grounded Reasoning (ICLR 2026 · similarity 0.93)
- TVI-CoT: Text-Visual Interleaved Chain-of-Thought Reasoning for Multimodal Understanding (ICML 2026 · similarity 0.92)
- UniVLR: Unifying Text and Vision in Visual Latent Reasoning for Multimodal LLMs (NeurIPS 2026 · similarity 0.92)
- SLVR: Structured Latent Visual Reasoning via Human-like Reasoning Flows (NeurIPS 2026 · similarity 0.91)
- No Labels, No Problem: Training Visual Reasoners with Multimodal Verifiers (ICLR 2026 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.