Research map / Vision-language and multimodal / Multimodal LLMs

RewardMap: Tackling Sparse Rewards in Fine-grained Visual Reasoning via Multi-Stage Reinforcement Learning

ICLR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Reinforcement learning for visual reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Builds ReasonMap-Plus, a dense-reward VQA extension of the ReasonMap transit-map benchmark, and proposes RewardMap, a multi-stage GRPO curriculum with difficulty-aware detail rewards that improves Qwen2.5-VL on transit-map reasoning and six general benchmarks.

Problem

Standard RL for fine-grained visual reasoning on high-resolution transit maps suffers from sparse, final-answer-only rewards and unstable optimization on hard route-planning tasks.

Approach

Adds partial-credit detail rewards for route planning (stops, route names, transfers, segments) scaled by map and question difficulty weights, and schedules RL stages from easy perception VQA (binary judgment, counting) to planning, replacing SFT cold-start with RL.

Key results

• RewardMap on Qwen2.5-VL-7B raises ReasonMap-Plus weighted accuracy from 44.21% to 74.25%, above Seed1.5-VL (73.58%).
• Average improvement of 3.47% across six general and spatial benchmarks (e.g., SpatialEval +13.51%) over Qwen2.5-VL-7B.
• Ablation: multi-stage design and reward design each add gains over the RL baseline (ReasonMap-Plus 67.61% to 73.12% and 71.07%, respectively).

Surprising findings

• SFT on ReasonMap-Plus followed by RL underperforms plain RL on the combined data for ReasonMap-Plus accuracy (60.53% vs 67.61%).

Limitations

• Gains on ReasonMap test accuracy remain well below closed-source GPT-5 (59.98%/62.50% vs 31.51%/31.77% for RewardMap on short/long), and planning accuracy still lags.
• Evaluation focuses on Qwen2.5-VL and one Kimi-VL model; generalization across architectures is only partially shown.
• Detail reward weights and difficulty weights are hand-designed with limited ablation range.

Experimental setup

Datasets: ReasonMap ReasonMap-Plus

Benchmarks: ReasonMap ReasonMap-Plus SEED-Bench-2-Plus SpatialEval V* Bench HRBench ChartQA MMStar

Metrics: weighted accuracy weighted map score accuracy

Baselines: GRPO baseline (format + correctness reward) SFT SFT → RL REINFORCE++ ReMax Seed1.5-VL GPT-4o GPT-5 Qwen2.5-VL-72B-Instruct Kimi-VL-A3B-Instruct

Builds on: GRPO ReasonMap REINFORCE++ ReMax LoRA

Closest papers by meaning

• ReasonMap: Towards Fine-Grained Visual Reasoning from Transit Maps (CVPR 2026 · similarity 0.92)
• Reason-RFT: Reinforcement Fine-Tuning for Visual Reasoning of Vision Language Models (NeurIPS 2025 · similarity 0.92)
• Open Vision Reasoner: Transferring Linguistic Cognitive Behavior for Visual Reasoning (NeurIPS 2025 · similarity 0.91)
• Perception-R1: Advancing Multimodal Reasoning Capabilities of MLLMs via Visual Perception Reward (ICLR 2026 · similarity 0.91)
• Pixel Reasoner: Incentivizing Pixel Space Reasoning via Curiosity-Driven Reinforcement Learning (NeurIPS 2025 · similarity 0.91)
• Activating Visual Context and Commonsense Reasoning Through Masked Prediction in VLMs (AAAI 2026 · similarity 0.91)