Research map / Large language models / Reasoning and chain of thought

On Robustness and Chain-of-Thought Consistency of RL-Finetuned VLMs

ICML 2026 · Large language models › Reasoning and chain of thought · approach: Theory of autoregressive reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Stress-tests RL-finetuned open-source and closed-source VLMs on simple spatial benchmarks with misleading captions and chain-of-thought prefixes, measuring accuracy, entropy and CoT-answer faithfulness, and studies how GRPO training with augmentation affects robustness and faithfulness.

Problem

RL-finetuned vision-language models reach high accuracy on visual reasoning benchmarks, but it is unclear whether they are robust to misleading textual context and whether their chain-of-thought stays consistent with their final answers.

Approach

Controlled textual perturbations (Stop-Think, Wrong-Think, Wrong-Caption, with optional disclaimers) applied to spatial VQA benchmarks, combined with entropy-based diagnostics and an RL training study that tests augmentation with correct and incorrect captions/thinking and a faithfulness reward.

Key results

• Under Wrong-Think, VLAA-Thinker's P(correct letter) collapses toward zero, while Stop-Think entropy rises for all open reasoning models.
• Closed-source models show markedly higher faithfulness than open-source models, with Gemini-3.1-Pro showing the strongest performance and robustness.
• Augmenting RL training with correct and wrong captions/thinking improves Wrong-Caption robustness, but faithfulness still drifts downward with RL training; judge agreement across three LLM judges is high (Fleiss' kappa about 0.81 to 0.88).

Surprising findings

• The models most robust to perturbations also show the highest inconsistency between CoT and answer even under clean Base prompts.
• Adding a correct caption or thinking string can reduce accuracy when paired with a disclaimer, mirroring the inverted effect of the disclaimer under wrong context.

Limitations

• Faithfulness is measured as reasoning-answer consistency via LLM judges, not internal causal faithfulness.
• Results show high seed variability in RL finetuning, limiting conclusions from single runs.
• Closed-source models were prompted to start from the wrong thinking string rather than forced by decoding, and some outputs were excluded.

Experimental setup

Datasets: SAT2 Pixmo-Count Geometry3K 3DSRBench CV-Bench Spatial-MM WhatsUp V*-Bench MME-RealWorld-Lite MMBench RefCOCOg

Benchmarks: 3DSRBench CV-Bench Spatial-MM WhatsUp V*-Bench MME-RealWorld-Lite MMBench RefCOCOg

Metrics: accuracy answer entropy P(correct letter) faithfulness rate abstention rate AUROC Cohen's kappa Fleiss' kappa

Baselines: SpaceR Video-R1 Vision-R1 VLAA-Thinker ViGoRL-Spatial o3 o4-mini Gemini-2.5-Pro Gemini-3.1-Pro Preview

Builds on: GRPO verl DeepSeek-R1

Closest papers by meaning

• Unveiling the Compositional Ability Gap in Vision-Language Reasoning Model (NeurIPS 2025 · similarity 0.92)
• Reinforcing Multimodal Reasoning Against Visual Degradation (NeurIPS 2026 · similarity 0.92)
• On the Faithfulness of Visual Thinking: Measurement and Enhancement (NeurIPS 2026 · similarity 0.92)
• Reason-RFT: Reinforcement Fine-Tuning for Visual Reasoning of Vision Language Models (NeurIPS 2025 · similarity 0.91)
• Fine-Tuning Large Vision-Language Models as Decision-Making Agents via Reinforcement Learning (NeurIPS 2024 · similarity 0.91)
• Activating Visual Context and Commonsense Reasoning Through Masked Prediction in VLMs (AAAI 2026 · similarity 0.91)