Research map / Vision-language and multimodal / Multimodal LLMs

Scaling Agentic Reinforcement Learning for Tool-Integrated Reasoning in VLMs

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Reinforcement learning for visual reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces VISTA-Gym, a Gymnasium-style environment with 26 visual tools and 7 VQA task families, and VISTA-R1, a VLM agent trained with SFT warmup followed by multi-turn GRPO to interleave reasoning with tool calls.

Problem

Open-source VLMs cannot reliably select, invoke, and coordinate visual tools for multi-step "thinking with images" reasoning, and no unified multimodal training environment exists for tool-integrated visual RL.

Approach

A scalable tool-integrated visual agentic training environment (VISTA-Gym) with standardized tool APIs, verifiable VQA tasks, and microservice-based tool serving, combined with a two-stage SFT-then-multi-turn-GRPO recipe (VISTA-R1) using format and repetition-penalized rule-based rewards.

Key results

• VISTA-R1-8B (InternVL3-8B) reaches 71.14% average accuracy across 11 VQA benchmarks, versus 57.49% for the base InternVL3-8B.
• VISTA-R1 outperforms similar-size baselines by 9.51%-18.72% with tools, per the abstract.
• Naively adding tools degrades InternVL3-8B average accuracy from 57.49% to 31.61%, while RL-trained VISTA-R1 with tools reaches 71.14%.

Surprising findings

• Directly exposing tools to base VLMs sharply degrades accuracy (e.g., GPT-5 from 75.84% to 51.03% average with tools), since tools act as distractors without instruction or reasoning priors.
• Commercial-model expert trajectories that hide intermediate reasoning (GPT-5) yield shorter traces and weaker gains from agentic RL than open-weights expert traces.

Limitations

• Rewards are sparse and terminal-only, so stepwise tool-use semantics are not directly rewarded.
• Evaluation uses benchmark-specific tool configurations for tool-integrated baselines, and most baselines are evaluated without tools, making comparisons less direct.
• Compute-heavy tools such as ChartMoE and G-LLaVA add infrastructure overhead.

Experimental setup

Datasets: FigureQA ChartQA Geometry3K GeoQA UniGeo MathVision MathV360 MapQA InfographicVQA ThinkVL VizWiz ScienceQA

Benchmarks: ChartQA Geometry3K GeoQA UniGeo MapQA TABMWP AI2D PlotQA CLEVR-Math IconQA MathVista

Metrics: accuracy

Baselines: GPT-5 Gemini-2.5-Pro Claude-4.5-Sonnet InternVL3-8B Qwen2.5-VL-7B VTool-R1-7B R1-VL-7B Perception-R1-7B

Builds on: ReAct GRPO Verl-Tool

Closest papers by meaning

• VAGEN: Reinforcing World Model Reasoning for Multi-Turn VLM Agents (NeurIPS 2025 · similarity 0.90)
• SenseSearch: Empowering Vision-Language Models with High-Resolution Agentic Search-Reasoning via Reinforcement Learning (CVPR 2026 · similarity 0.90)
• VTool-R1: VLMs Learn to Think with Images via Reinforcement Learning on Multimodal Tool Use (ICLR 2026 · similarity 0.90)
• Fine-Tuning Large Vision-Language Models as Decision-Making Agents via Reinforcement Learning (NeurIPS 2024 · similarity 0.90)
• Agentic Jigsaw Interaction Learning for Enhancing Visual Perception and Reasoning in Vision-Language Models (ICLR 2026 · similarity 0.90)
• Learning to Select Visual Tools from Experience (CVPR 2026 · similarity 0.90)