Research map / Vision-language and multimodal / Multimodal LLMs

Semi-off-Policy Reinforcement Learning for Vision-Language Slow-Thinking Reasoning

NeurIPS 2025 · Vision-language and multimodal › Multimodal LLMs · approach: Reinforcement learning for visual reasoning

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes SOPHIA, a semi-off-policy RL method that pairs on-policy LVLM image captions with off-policy LLM reasoning trajectories and propagates outcome rewards back to captions to train InternVL slow-thinking reasoning.

Problem

On-policy RL for large vision-language models is bounded by the initial policy's rollout space, since LVLMs lack slow-thinking trajectories, while naive off-policy distillation from external models causes visual hallucinations due to perception mismatch.

Approach

Build a semi-off-policy behavior model: the trainable LVLM generates image descriptions (on-policy visual understanding), and an external reasoning LLM solves queries conditioned on those descriptions (off-policy reasoning). Outcome rewards from reasoning are averaged back to each caption as a visual reward, and only correct, high-visual-reward, shortest trajectories are used for importance-weight

Key results

• InternVL3.0-38B + SOPHIA reaches 49.08% on MathVision and 49.95% on OlympiadBench pass@1, with average 55.51 vs 47.01 for the base model across the eight benchmarks.
• InternVL2.5-38B + SOPHIA averages 54.03 vs 43.09 base and 47.74 for MPO; ablation on InternVL2.5-38B gives 55.13 with the full reward.
• SOPHIA+GRPO on InternVL3.0-38B reaches average 57.28, and SOPHIA avoids the performance drop seen with direct GRPO over 480 steps.

Measured effects reported in the paper

Surprising findings

• Random sampling of off-policy trajectories is a strong baseline, gaining notably over the base model on MathVision and OlympiadBench.
• Adding captions as training data gives no benefit and prepending captions to trajectories can hurt performance.
• SOPHIA underperforms the base on MathVista at 8B scale, attributed to limited coverage of problem types.

Limitations

• Inference is about 3-4 times slower than the base LVLM due to long slow-thinking outputs.
• Persistent hallucination and reasoning redundancy; weak long-range visual dependency and fine-grained recognition.
• Training data is private and not released, and the method relies on reasoning-model teachers (QwQ, DeepSeek-R1).

Experimental setup

Datasets: private VQA training set (~80K) RLAIF-V WildVision MathV360K Numina MATH AMC/AIME (excluding AIME 2024)

Benchmarks: MMMU MMMU Pro MathVista MathVerse DynaMath MathVision MV-MATH OlympiadBench

Metrics: pass@1 accuracy average accuracy

Baselines: GPT-4.1 Claude-3.7-Sonnet InternVL2.5-38B-MPO InternVL3.0-38B-MPO InternVL3.0-38B-GRPO MM-Eureka-Qwen-32B Qwen2.5-VL-72B QvQ-72B-preview Virgo-72B Mulberry-8B MiniCPM-o-8B MM-Eureka-8B

Builds on: GRPO DeepSeek-R1 QwQ-32B-Preview InternVL MPO

Closest papers by meaning

• More Thought, Less Accuracy? On the Dual Nature of Reasoning in Vision-Language Models (ICLR 2026 · similarity 0.92)
• VL-Rethinker: Incentivizing Self-Reflection of Vision-Language Models with Reinforcement Learning (NeurIPS 2025 · similarity 0.91)
• Think or Not? Selective Reasoning via Reinforcement Learning for Vision-Language Models (NeurIPS 2025 · similarity 0.91)
• All Roads Lead to Rome: Incentivizing Divergent Thinking in Vision-Language Models (CVPR 2026 · similarity 0.91)
• PeRL: Permutation-Enhanced Reinforcement Learning for Interleaved Vision-Language Reasoning (NeurIPS 2025 · similarity 0.90)
• Unveiling Chain of Step Reasoning for Vision-Language Models with Fine-grained Rewards (NeurIPS 2025 · similarity 0.90)