Research map / Vision-language and multimodal / Multimodal LLMs

ARM-Thinker: Reinforcing Multimodal Generative Reward Models with Agentic Tool Use and Visual Reasoning

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Reinforcement learning for visual reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes ARM-Thinker, an agentic multimodal reward model that invokes image-crop, document-retrieval and instruction-check tools inside a think-act-observe loop, trained with SFT cold start and two-stage GRPO on Qwen2.5-VL-7B, and introduces the ARMBench-VL benchmark.

Problem

Multimodal reward models hallucinate, ground weakly in visual evidence, and cannot use tools to verify fine-grained details, multi-page documents, or instruction constraints during judgment.

Approach

Turn the generative reward model into an agent that alternates thinking and tool calls (zoom-in crop, document page retrieval, instruction validators), trained with a two-stage GRPO reward that first encourages tool exploration and then rewards accuracy plus successful tool use.

Key results

• ARM-Thinker-7B reaches 67.8% on VL-RewardBench overall (+17.7 over Qwen2.5-VL-7B) and 64.6% on ARMBench-VL (+18.5), averaging +16.2 across reward benchmarks.
• On think-with-images benchmarks ARM-Thinker-7B averages 76.5%, +9.6 over Qwen2.5-VL-7B and comparable to Mini-o3 (76.1%).
• On six general math/logic reasoning benchmarks it averages 49.0%, +4.2 over Qwen2.5-VL-7B, with +10.9 on WeMath and +8.7 on LogicVista.

Surprising findings

• Qwen2.5-VL-7B performance degrades when tools are enabled (e.g., V* drops from 75.4 to 50.3), while ARM-Thinker improves with tools.
• Accuracy-only and fixed tool-bonus rewards both underperform the adaptive tool reward, with a fixed bonus causing overuse of tool calls.

Limitations

• Benchmark ARMBench-VL is small (1,499 questions) and partly built with proprietary or large-model-generated data.
• Instruction-following tool-use data was excluded from RL training, relying on generalization that is not deeply analyzed.
• The paper lacks a clearly specified adaptive reward formula and compute/hardware details.

Experimental setup

Datasets: LLaVA-Critic DeepEyes MM-IFEngine MP-DocVQA V*Bench MMlongbench-doc MM-IFEval

Benchmarks: VL-RewardBench RewardBench-2 ARMBench-VL V*Bench HRBench-4K HRBench-8K MME-RealWorld MMMU MathVista MathVision MathVerse WeMath

Metrics: accuracy

Baselines: Qwen2.5-VL-7B Qwen3-VL-8B InternVL3-8B InternVL3.5-8B GPT-4o UnifiedReward-7B DeepEyes Pixel Reasoner Mini-o3 Gemma-3-27B

Builds on: ReAct GRPO WebWatcher DeepEyes MM-IFEngine LLaVA-Critic

Closest papers by meaning

• VL-RewardBench: A Challenging Benchmark for Vision-Language Generative Reward Models (CVPR 2025 · similarity 0.90)
• Agent0-VL: Exploring Self-Evolving Agent for Tool-Integrated Vision-Language Reasoning (ICML 2026 · similarity 0.90)
• DeepEyesV2: Toward Agentic Multimodal Model (ICLR 2026 · similarity 0.90)
• Bad Seeing or Bad Thinking? Rewarding Perception for Multimodal Reasoning (ICML 2026 · similarity 0.89)
• Unified Multimodal Chain-of-Thought Reward Model through Reinforcement Fine-Tuning (NeurIPS 2025 · similarity 0.89)
• ThinkMorph: Emergent Properties in Multimodal Interleaved Chain-of-Thought Reasoning (ICLR 2026 · similarity 0.89)