Research map / Large language models / Reasoning and chain of thought

SophiaVL-R1: Reinforcing MLLMs Reasoning with Thinking Reward

ICLR 2026 · Large language models › Reasoning and chain of thought · approach: RL with verifiable rewards for reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes SophiaVL-R1, which adds a holistic thinking reward from a trained 3B reward model to rule-based outcome-reward GRPO, using a trustworthiness-weighted and time-annealed reward (Trust-GRPO) to train a 7B MLLM.

Problem

Outcome-only rule-based RL for multimodal LLMs does not supervise the reasoning process, so models can reach correct answers via flawed reasoning and may generalize poorly; model-generated thinking rewards are also prone to reward hacking.

Approach

Train a holistic thinking reward model on GRPO rollouts scored by Qwen2.5-VL-72B, then integrate its reward into GRPO with a trustworthiness weight (from mean thinking reward gap between correct and incorrect response groups) and an exponential decay over training steps.

Key results

• SophiaVL-R1-7B reaches 71.3 on MathVista, above Qwen2.5-VL-7B+GRPO (69.9) and LLaVA-OneVision-72B (68.4).
• On MathVerse, SophiaVL-R1-7B scores 48.8 versus 30.7 for InternVL2.5-8B-VisualPRM.
• On MMMU, SophiaVL-R1-7B scores 61.3, 4.5 points above LLaVA-OneVision-72B (56.8).

Surprising findings

• An untrained thinking reward model (Qwen2.5-VL-3B-Instruct) gives performance comparable to plain GRPO, indicating the trained reward model is essential.
• Directly combining thinking and outcome rewards without trust weighting or annealing performs worse than GRPO on several benchmarks.

Limitations

• Thinking reward labels come from a single 72B MLLM judge, and the reward model is trained on rollouts from the same policy family.
• Trust weight and decay schedule are heuristic; the paper notes learned or reward-gated schedules are left for future work.
• Evaluation on several benchmarks relies on baseline numbers taken from leaderboards and differing evaluation settings.

Experimental setup

Datasets: SophiaVL-R1-130k SophiaVL-R1-Thinking-156k MathVista MathVerse MMMU MME MMStar ChartQA MMBench VLRewardBench

Benchmarks: MathVista MathVerse MMMU MME MMStar ChartQA MMBench VLRewardBench

Metrics: accuracy average score

Baselines: Qwen2.5-VL-7B-Instruct +GRPO Qwen2.5-VL-7B-Instruct +SFT+GRPO LLaVA-OneVision-72B LLaVA-OneVision-7B R1-OneVision-7B Curr-ReFT-7B URSA-8B InternVL2.5-8B-VisualPRM Math-PUMA-Qwen2VL-7B Multimath-7B Math-LLaVA-13B Cambrian-1-34B

Builds on: GRPO DeepSeek-R1 Qwen2.5-VL VeRL

Closest papers by meaning

• RM-R1: Reward Modeling as Reasoning (ICLR 2026 · similarity 0.91)
• Think-RM: Enabling Long-Horizon Reasoning in Generative Reward Models (NeurIPS 2025 · similarity 0.90)
• R-4B: Incentivizing General-Purpose Auto-Thinking in MLLMs via Bi-Mode Annealing and Reinforce Learning (CVPR 2026 · similarity 0.90)
• Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMs (ICLR 2026 · similarity 0.90)
• Beyond Binary Rewards: Training LMs to Reason About Their Uncertainty (ICLR 2026 · similarity 0.90)
• Co-rewarding: Stable Self-supervised RL for Eliciting Reasoning in Large Language Models (ICLR 2026 · similarity 0.90)