Research map / Vision-language and multimodal / Multimodal LLMs
Unveiling Chain of Step Reasoning for Vision-Language Models with Fine-grained Rewards
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Chain-of-Step (CoS) reasoning for VLMs, with structured step-level SFT data (ShareGPT-Step-300K), a process reward model, and PRM-guided iterative DPO and step-level beam search, improving LLaVA-NeXT and InternVL-2.5-MPO on multimodal reasoning benchmarks.
Problem
Chain-of-thought reasoning for vision-language models is coarse-grained and unstructured, making it hard to evaluate intermediate reasoning quality and select good reasoning paths for RL training or inference-time scaling.
Approach
Decompose VLM reasoning into structured steps (Name, Thought, Reflection) delimited by special tokens, train a process reward model on Monte Carlo and GPT-4o step labels, and use PRM scores for iterative DPO (weighted step+answer score) and step-level beam search.
Key results
- CoS on InternVL2.5-MPO (8B) reaches 73.4 average across six benchmarks, versus 69.4 for the base model and 71.0 after SFT.
- CoS-LLaVA improves LLaVA-NeXT-LLaMA3 average from 52.1 to 64.2 with iterative DPO.
- In the GRPO ablation, PRM-based step-and-answer rewards reach 63.0 average versus 61.2 for outcome rewards.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add iterative DPO after SFT | LLaVA-NeXT-LLaMA3 · Average across six benchmarks | score: 60.8 -> 64.2 | Helped |
| Add iterative DPO after SFT | InternVL2.5-MPO · Average across six benchmarks | score: 71.0 -> 73.4 | Helped |
| Use PRM scores for answers instead of outcome-based selection (ablation) | LLaVA-NeXt-SFT with iterative DPO · MathVista | score: 53.5 -> 53.1 | Hurt |
| Use PRM scores for answers instead of outcome-based selection (ablation) | LLaVA-NeXt-SFT with iterative DPO · MMStar | score: 58.1 -> 57.3 | Hurt |
| Use PRM scores for answers instead of outcome-based selection (ablation) | LLaVA-NeXt-SFT with iterative DPO · M3CoT | score: 70.0 -> 69.7 | Hurt |
| Add PRM scores for reasoning steps to answer-quality scoring (ablation) | LLaVA-NeXt-SFT with iterative DPO · MathVista | score: 53.5 -> 54.7 | Helped |
| Add PRM scores for reasoning steps to answer-quality scoring (ablation) | LLaVA-NeXt-SFT with iterative DPO · MMStar | score: 58.1 -> 58.9 | Helped |
| Add PRM scores for reasoning steps to answer-quality scoring (ablation) | LLaVA-NeXt-SFT with iterative DPO · M3CoT | score: 70.0 -> 71.7 | Helped |
| Use a 38B rather than 8B PRM base model (ablation) | PRM · Unseen questions | Step Acc: 85.7 -> 87.3 | Helped |
| Use CoS structured reasoning and PRM rewards rather than direct-prompt reasoning and outcome rewards (ablation) | LLaVA-NeXT · M3CoT | score: 69.3 -> 71.7 | Helped |
| Use CoS structured reasoning and PRM rewards rather than direct-prompt reasoning and outcome rewards (ablation) | LLaVA-NeXT · MathVista | score: 53.1 -> 54.7 | Helped |
| Use CoS structured reasoning and PRM rewards rather than direct-prompt reasoning and outcome rewards (ablation) | LLaVA-NeXT · MMStar | score: 58.2 -> 58.9 | Helped |
Surprising findings
- Per-step DPO significantly hurt performance (M3CoT 67.4 to 64.8), as chosen and rejected steps were too similar, causing both rewards to become negative.
- The larger 38B PRM achieved higher step accuracy but not higher answer accuracy than 8B PRMs.
Limitations
- Process labels from Monte Carlo estimation and GPT-4o judging are not guaranteed to be 100% correct.
- No human annotation of process data due to cost and scalability.
- Step-wise DPO attempt failed, indicating the step-level preference construction is not yet effective.
Experimental setup
Datasets: ShareGPT-Step-300K M3CoT MathVista MMStar MMMU ChartQA AI2D GeoQA+ MathVision ScienceQA DocVQA InfoVQA
Benchmarks: MathVista MMStar MMMU M3CoT AI2D ChartQA
Metrics: accuracy average accuracy best-of-N accuracy pass@N step accuracy answer accuracy
Baselines: LLaVA-NeXT-LLaMA3 SFT InternVL2.5-MPO LLaVA-CoT Insight-V Self-Consistency outcome reward DPO Direct Prompt step-DPO
Builds on: chain-of-thought Math-Shepherd iterative DPO InternVL-2.5-MPO LLaVA-NeXT
Closest papers by meaning
- Latent Chain-of-Thought for Visual Reasoning (NeurIPS 2025 · similarity 0.92)
- HopChain: Multi-Hop Data Synthesis for Generalizable Vision-Language Reasoning (NeurIPS 2026 · similarity 0.92)
- Let VLMs Grade Their Own Thoughts: A Self-Quantification Approach to Reasoning-Aware Reward Modeling (CVPR 2026 · similarity 0.91)
- Fine-Tuning Large Vision-Language Models as Decision-Making Agents via Reinforcement Learning (NeurIPS 2024 · similarity 0.91)
- More Thought, Less Accuracy? On the Dual Nature of Reasoning in Vision-Language Models (ICLR 2026 · similarity 0.91)
- OpenVLThinker: Complex Vision-Language Reasoning via Iterative SFT-RL Cycles (NeurIPS 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.