Research map / Vision-language and multimodal / Multimodal LLMs

All Roads Lead to Rome: Incentivizing Divergent Thinking in Vision-Language Models

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Reinforcement learning for visual reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes MUPO, a GRPO drop-in that clusters sampled responses into groups, computes group-local advantages, and adds a diversity reward to keep divergent reasoning strategies alive in VLM RL training.

Problem

GRPO-based RL for vision-language models causes diversity collapse: reasoning strategies converge early, reducing exploration and test-time scaling (acc@k) relative to base models.

Approach

Partition sampled responses into multiple embedding-based groups, apply GRPO-style advantages locally within each group with load-balance weights, and add an annealed, accuracy-gated cosine-distance diversity reward between groups.

Key results

• MUPO-Thinker-7B reaches average acc@1 51.6 on math benchmarks vs 49.1 for prior best (Vision-R1-7B), and acc@4 58.8 vs 52.8.
• On general benchmarks MUPO-Thinker-7B averages acc@1 65.6 vs 63.6 (VLAA) and acc@4 72.4 vs 71.5 (QwenVL).
• MUPO-Thinker-3B averages acc@1 43.5 on math benchmarks vs 41.5 for LMM-R1 and acc@4 51.5 vs 50.1 for LMM-R1 (3B-scale comparison in table).

Surprising findings

• Base Qwen2.5-VL models outperform RL-trained models under multiple sampling (acc@k), despite RL models winning at acc@1.
• Reasoning diversity correlates strongly with acc@4 across base model samples.

Limitations

• Diversity reward depends on an embedding model and clustering choices, adding design complexity.
• Gains over baselines are modest on some benchmarks (e.g., acc@1 improvements of about 2 to 2.5 points).

Experimental setup

Datasets: ViRL39K

Benchmarks: MathVerse MathVista MathVision LogicVista WeMath Geometry3k MMStar HallusionBench MMVet

Metrics: acc@1 (greedy) acc@4 (4 samples at temperature 1.0) acc@k pairwise reasoning diversity

Baselines: Vision-R1-7B VLAA-Thinker R1-OneVision-7B InternVL2.5-8B VLM-R1 LMM-R1 GPT-5-Thinking Gemini-2.5-Pro

Builds on: GRPO Vision-R1 Qwen2.5-VL

Closest papers by meaning

• Think or Not? Selective Reasoning via Reinforcement Learning for Vision-Language Models (NeurIPS 2025 · similarity 0.94)
• More Thought, Less Accuracy? On the Dual Nature of Reasoning in Vision-Language Models (ICLR 2026 · similarity 0.91)
• Semi-off-Policy Reinforcement Learning for Vision-Language Slow-Thinking Reasoning (NeurIPS 2025 · similarity 0.91)
• VL-Rethinker: Incentivizing Self-Reflection of Vision-Language Models with Reinforcement Learning (NeurIPS 2025 · similarity 0.90)
• Fast-Slow Thinking GRPO for Large Vision-Language Model Reasoning (NeurIPS 2025 · similarity 0.90)
• Fine-Tuning Large Vision-Language Models as Decision-Making Agents via Reinforcement Learning (NeurIPS 2024 · similarity 0.89)