Research map / Vision-language and multimodal / Multimodal LLMs
Grounded Reinforcement Learning for Visual Reasoning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Trains VLMs (Qwen2.5-VL-3B/7B) to produce coordinate-grounded reasoning steps via MCTS-built SFT warm start, GRPO, and multi-turn zoom-in tool-call RL, improving spatial reasoning, GUI grounding, and visual search.
Problem
Vision-language models reason in ungrounded, abstract ways, and standard RL on VLMs amplifies this rather than inducing visual reasoning behaviors such as region exploration, visual subgoaling, verification, and backtracking.
Approach
Each reasoning step is a tuple of a thought and an (x,y) image coordinate; MCTS stitches teacher-sampled grounded steps into warm-start traces, followed by GRPO with format rewards and a multi-turn variant where the model requests zoomed crops at predicted coordinates.
Key results
- ViGoRL-3B reaches 62.9% on SAT-2 val vs 50.0% for vanilla GRPO (+12.9) and 46.1% for base Qwen2.5-VL-3B.
- Multi-turn ViGoRL-7B reaches 86.4% on V*Bench, above GPT-4o (66.0%) and Sketchpad-GPT-4o (80.3%).
- ViGoRL-7B scores 91.0% on ScreenSpot-V2 and 33.1% on ScreenSpot-Pro; human study finds 72.8% of predicted coordinates accurately refer to the described region.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add grounded reasoning with MCTS warm-start and GRPO | Qwen2.5-VL-3B · SAT-2 | accuracy: 50.0 -> 62.9; +12.9 points over Vanilla GRPO | Helped |
| Add grounded reasoning with MCTS warm-start and GRPO | Qwen2.5-VL-3B · BLINK | accuracy: 46.5 -> 48.5; 2.0 points | Helped |
| Add grounded reasoning with MCTS warm-start and GRPO | Qwen2.5-VL-3B · RoboSpatial | accuracy: 69.7 -> 67.1 | Hurt |
| Add multi-turn RL with zoomed visual feedback | ViGoRL-3B · ScreenSpot-Pro | accuracy: 31.1% -> 32.3%; 1.2 percentage point improvement | Helped |
| Add multi-turn RL with visual feedback | ViGoRL-3B · V*Bench | accuracy: 79.1 -> 81.2 | Helped |
| Add GRPO refinement (ablation) | ViGoRL with MCTS warm-start and grounding · SAT-2 | accuracy: 58.83 { (±1.53)} -> 62.93 { (±1.50)}; (–4.10%) | Helped |
| Add GRPO refinement (ablation) | ViGoRL with MCTS warm-start and grounding · BLINK | accuracy: 44.97 { (±2.32)} -> 48.50 { (±2.33)}; (–3.53%) | Helped |
| Add MCTS warm-start traces (ablation) | ViGoRL with GRPO and grounding · SAT-2 and BLINK | valid output formatting | Helped |
| Add explicit spatial grounding (ablation) | ViGoRL with MCTS warm-start and GRPO · SAT-2 | accuracy: 58.69 { (±1.53)} -> 62.93 { (±1.50)}; (–4.28%) | Helped |
| Add explicit spatial grounding (ablation) | ViGoRL with MCTS warm-start and GRPO · BLINK | accuracy: 45.44 { (±2.32)} -> 48.50 { (±2.33)}; (–3.06%) | Helped |
| Use MCTS search instead of successful teacher linear rollouts (ablation) | ViGoRL with grounded warm-start and GRPO · BLINK | accuracy: 46.18 { (±2.32)} -> 48.50 { (±2.33)}; (–2.32%) | Helped |
| Add direct-answer SFT pretraining (ablation) | ViGoRL warm-start training and GRPO · SAT-2 and BLINK | accuracy: 62.93 -> 63.26; 48.50 -> 48.56; <1 point | No clear effect |
Surprising findings
- Vanilla GRPO on Qwen2.5-VL-3B drops visual subgoal setting to 0.00 and does not increase backtracking, showing standard RL exacerbates ungrounded reasoning.
- Without MCTS, distillation from linear teacher rollouts matches in-distribution SAT-2 accuracy but degrades BLINK out-of-distribution performance.
Limitations
- Final-answer-only reward allows reward hacking with partially incorrect reasoning.
- Model often generates long reasoning chains even for simple questions.
- Only point-based crops and a limited set of visual tools are explored.
Experimental setup
Datasets: SAT-2 OS-Atlas ICAL SA-1B (Segment Anything)
Benchmarks: SAT-2 BLINK RoboSpatial ScreenSpot-V2 ScreenSpot-Pro VisualWebArena V*Bench
Metrics: accuracy grounding hit rate task success rate visual behavior counts
Baselines: SFT direct Vanilla GRPO Qwen2.5-VL-72B GPT-4o Sketchpad-GPT-4o ICAL OS-Atlas UI-TARS
Builds on: GRPO DeepSeek-R1 STaR Visual-RFT ViperGPT
Closest papers by meaning
- iVGR: Internalizing Visually Grounded Reasoning for MLLMs with Reinforcement Learning (ICML 2026 · similarity 0.91)
- GRIT: Teaching MLLMs to Think with Images (NeurIPS 2025 · similarity 0.90)
- Start Small, Think Big: Curriculum-based Relative Policy Optimization for Visual Grounding (AAAI 2026 · similarity 0.90)
- RewardMap: Tackling Sparse Rewards in Fine-grained Visual Reasoning via Multi-Stage Reinforcement Learning (ICLR 2026 · similarity 0.90)
- DeepEyes: Incentivizing "Thinking with Images" via Reinforcement Learning (ICLR 2026 · similarity 0.90)
- No Labels, No Problem: Training Visual Reasoners with Multimodal Verifiers (ICLR 2026 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.