Research map / Reinforcement learning / RL from feedback
Enhancing the Outcome Reward-based RL Training of MLLMs with Self-Consistency Sampling
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Self-Consistency Sampling (SCS), which adds a consistency reward from truncation-resampling and visual perturbation to outcome-reward RL (RLOO, GRPO, REINFORCE++) to penalize unfaithful reasoning in multimodal multiple-choice training.
Problem
Outcome-reward RL for MLLMs on multiple-choice tasks rewards guessed correct answers after flawed reasoning the same as faithful reasoning, producing unfaithful trajectories.
Approach
Adds a differentiable consistency reward computed from answer agreement across truncated-and-resampled continuations under small Gaussian image perturbations, down-weighting unreliable traces during policy updates without a reward model.
Key results
- RLOO with SCS on Qwen2.5-VL-7B-Instruct reaches 65.5 overall vs 57.8 for RLOO baseline (+7.7 points).
- GRPO with SCS improves overall from 63.6 to 64.5 (+0.9); REINFORCE++-baseline +1.7 and REINFORCE++ +2.0.
- Quantitative analysis shows about 14-15% fewer unfaithful reasoning cases among correctly answered questions after SCS, per human and LLM judges.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add SCS to RLOO | Qwen2.5-VL-7B-Instruct · Overall | accuracy: 57.8 -> 65.5 (+7.7) | Helped |
| Add SCS to GRPO | Qwen2.5-VL-7B-Instruct · Overall | accuracy: 63.6 -> 64.5 (+0.9) | Helped |
| Add SCS to REINFORCE++-baseline | Qwen2.5-VL-7B-Instruct · Overall | accuracy: 61.3 -> 63.0 (+1.7) | Helped |
| Add SCS to REINFORCE++ | Qwen2.5-VL-7B-Instruct · Overall | accuracy: 60.9 -> 62.9 (+2.0) | Helped |
| Add SCS to RLOO | Qwen2.5-VL-3B-Instruct · Overall | accuracy: 54.7 -> 57.9 (+3.2) | Helped |
| Add SCS to RLOO | InternVL3-8B · Overall | accuracy: 61.7 -> 63.3 (+1.6) | Helped |
| Add SCS to RLOO | Qwen2.5-VL-7B-Instruct · ScienceQA | accuracy: 53.9 -> 68.8 | Helped |
| Add SCS to RLOO | Qwen2.5-VL-7B-Instruct · MathVerse | accuracy: 56.8 -> 67.1 | Helped |
| Add SCS to RLOO | Qwen2.5-VL-7B-Instruct · M3CoT | accuracy: 67.6 -> 75.7 | Helped |
| Add SCS to RLOO | Qwen2.5-VL-7B-Instruct · WeMath | accuracy: 86.4 -> 88.1 | Helped |
| Add SCS to RLOO | Qwen2.5-VL-7B-Instruct · MMMU-val | accuracy: 51.5 -> 59.1 | Helped |
| Add SCS to RLOO | Qwen2.5-VL-7B-Instruct · MathVision | accuracy: 30.4 -> 34.0 | Helped |
Surprising findings
- Truncation-resampling and visual perturbation each alone yield around +5 points, while combining them yields +7.7 for RLOO.
- SFT on the same data produces responses with no reasoning, skipping intermediate rationale.
Limitations
- Method has not been extensively applied to text-only LLMs or a broader range of MLLMs.
- Evaluated only on multiple-choice questions; unfaithfulness measure relies on manual and LLM judgments on 100 samples per benchmark.
- Consistency reward depends on assumptions (unique correct trajectory, leaf-choice alignment) that are argued rather than fully verified.
Experimental setup
Datasets: M3CoT Geometry3K ScienceQA
Benchmarks: M3CoT MMMU-val ScienceQA WeMath MathVerse MathVision
Metrics: accuracy unfaithful reasoning count (human and LLM judged)
Baselines: SFT GRPO REINFORCE++-baseline REINFORCE++ RLOO
Builds on: GRPO RLOO REINFORCE++ PPO
Closest papers by meaning
- Consistent Paths Lead to Truth: Self-Rewarding Reinforcement Learning for LLM Reasoning (NeurIPS 2025 · similarity 0.91)
- Stable and Efficient Single-Rollout RL for Multimodal Reasoning (CVPR 2026 · similarity 0.91)
- Co-rewarding: Stable Self-supervised RL for Eliciting Reasoning in Large Language Models (ICLR 2026 · similarity 0.91)
- LaSeR: Reinforcement Learning with Last-Token Self-Rewarding (ICLR 2026 · similarity 0.90)
- Reshaping Reasoning in LLMs: A Theoretical Analysis of RL Training Dynamics through Pattern Selection (ICLR 2026 · similarity 0.90)
- SophiaVL-R1: Reinforcing MLLMs Reasoning with Thinking Reward (ICLR 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.