Research map / Vision-language and multimodal / Multimodal LLMs
GRIT: Teaching MLLMs to Think with Images
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes GRIT, which trains MLLMs with RL (GRPO-GR) to produce reasoning chains interleaving natural language and bounding boxes, using only 20 image-question-answer triplets.
Problem
Open-source visual reasoning models produce pure natural-language reasoning that is weakly grounded in image regions, and human-annotated grounded reasoning data is scarce.
Approach
Grounded reasoning paradigm where the model interleaves text with bounding-box coordinates (without pixel re-injection), trained by GRPO-GR with format rewards (think/rethink tokens, bbox syntax), a counting reward, and a GPT-4o answer-accuracy reward, requiring no reasoning or box annotations.
Key results
- GRIT on Qwen2.5-VL 3B reaches 72.9 ACC on VSR (vs 49.5 direct query) and 0.325 GIoU on VSR, with 20 training samples.
- GRIT on InternVL3 2B reaches 64.9 ACC on VSR and 44.2 ACC on TallyQA, improving over few-shot SFT (54.0 and 22.5).
- Attention to image tokens is higher in rethink segments when pre-rethink boxes are present, indicating boxes boost visual attention in later reasoning.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| train with GRIT instead of few-shot fine-tuning | Qwen2.5-VL-3B · VSR | ACC: 59.7 -> 72.9 | Helped |
| train with GRIT instead of few-shot fine-tuning | Qwen2.5-VL-3B · TallyQA | ACC: 44.5 -> 47.8 | Helped |
| train with GRIT instead of few-shot fine-tuning | Qwen2.5-VL-3B · GQA | ACC: 64.6 -> 62.8 | Hurt |
| train with GRIT instead of direct query | Qwen2.5-VL-3B · MathVista | ACC: 58.5 -> 59.8 | Helped |
| train with GRIT instead of direct query | Qwen2.5-VL-3B · MME | ACC: 88.9 -> 89.3 | Helped |
| train with GRIT instead of few-shot fine-tuning | InternVL3-2B · VSR | ACC: 54.0 -> 64.9 | Helped |
| train with GRIT instead of few-shot fine-tuning | InternVL3-2B · TallyQA | ACC: 22.5 -> 44.2 | Helped |
| train with GRIT instead of few-shot fine-tuning | InternVL3-2B · GQA | ACC: 46.8 -> 63.2 | Helped |
| train with GRIT instead of few-shot fine-tuning | InternVL3-2B · MathVista | ACC: 17.4 -> 48.2 | Helped |
| train with GRIT instead of direct query | InternVL3-2B · MME | ACC: 40.0 -> 82.0 | Helped |
| remove counting-related training data and grounded-target-counting reward (ablation) | GRIT on InternVL3-2B · In-domain | GIoU: 0.387 -> 0.349 | Hurt |
| remove counting-related training data and grounded-target-counting reward (ablation) | GRIT on InternVL3-2B · Out-of-domain | GIoU: 0.437 -> 0.378 | Hurt |
Surprising findings
- Zero-shot CoT and one-shot ICL severely degrade answer accuracy on several benchmarks (e.g., InternVL3 CoT 6.4 ACC on VSR vs 52.9 direct query), indicating grounding and reasoning interfere without training.
- Scaling training data from 20 to 7,000 samples yields diminishing gains and small out-of-domain improvements, suggesting data diversity matters more than volume.
Limitations
- Evaluated only on small models (Qwen2.5-VL-3B and InternVL3-2B) due to limited resources.
- Out-of-domain generalization is limited; gains saturate with more training data.
- Relies on GPT-4o as judge for answer-accuracy reward and evaluation, adding API dependence.
Experimental setup
Datasets: VSR TallyQA GQA MME MathVista OVDEval Visual CoT
Benchmarks: VSR TallyQA GQA MathVista MME OVDEval
Metrics: GPT-as-judge answer accuracy grounding IoU vision-language reasoning cross-modal correlation
Baselines: Direct query Chain-of-Thought One-shot ICL Few-shot fine-tuning VLM-R1
Builds on: GRPO DeepSeek-R1 VLM-R1 Visual CoT
Closest papers by meaning
- iVGR: Internalizing Visually Grounded Reasoning for MLLMs with Reinforcement Learning (ICML 2026 · similarity 0.91)
- Grounded Reinforcement Learning for Visual Reasoning (NeurIPS 2025 · similarity 0.90)
- VLM-R³: Region Recognition, Reasoning, and Refinement for Enhanced Multimodal Chain-of-Thought (NeurIPS 2025 · similarity 0.90)
- Multi-step Visual Reasoning with Visual Tokens Scaling and Verification (NeurIPS 2025 · similarity 0.90)
- VTool-R1: VLMs Learn to Think with Images via Reinforcement Learning on Multimodal Tool Use (ICLR 2026 · similarity 0.90)
- VGR: Visual Grounded Reasoning (ICLR 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.