Research map / Large language models / Reasoning and chain of thought

Monitorability as a Free Gift: How RLVR Spontaneously Aligns Reasoning

ICML 2026 · Large language models › Reasoning and chain of thought · approach: Long CoT dynamics and efficiency · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Systematically studies how monitorability (CoT faithfulness to hint interventions, measured by g-mean²) evolves under vanilla RLVR across math, code, science, and instruction-following data, finding gains are data-dependent, orthogonal to capability, and linked to entropy reduction and prompt-directed attention.

Problem

Whether monitorability of chain-of-thought in large reasoning models, which can emerge as a "free gift" early in RLVR, generalizes across training domains, tracks capability, and what mechanisms drive it.

Approach

Empirical, controlled study of monitorability dynamics under vanilla RLVR (no monitor in the reward) across domain-specific data mixtures, training length and difficulty, with mechanistic analysis via response entropy and cross-segment attention; proposes an IF Cascade strategy that boosts monitorability after task-specific RLVR.

Key results

• Monitorability gains are strongly data-dependent: IF-containing training (IF, ALL) gives the largest early gains (IF +0.17, ALL +0.15 g-mean² vs. Math +0.11 and Sci +0.06 in 0-300 steps), while late-phase gains are near zero or negative (Math -0.04).
• The IF Cascade (+IF) strategy reaches peak average g-mean² of 0.639 on Qwen3-4B and 0.566 on DS1.5B, above ALL (0.633 and 0.500).
• Capability correlates weakly and inconsistently with monitorability; for Qwen3-4B, Math monitorability correlates negatively with capability, and training on harder problems (Hard) lowers average monitorability to 0.21 vs 0.39 for Easy.

Surprising findings

• Monitorability is orthogonal to or negatively correlated with capability; the IF-only model shows substantial performance degradation on DS1.5B despite high monitorability.
• Attention from the answer to the reasoning trace (A→R) is negatively correlated with monitorability, while attention to the prompt is the strongest positive predictor.

Limitations

• Monitorability is measured via hint-based interventions and a single primary monitor (Qwen2.5-32B-Instruct), so a high score may reflect prompt sensitivity (a 'false sense of monitorability')
• Mechanistic findings are correlational; entropy and attention correlations do not establish causation
• Only two base models and a single vanilla RLVR setup were studied

Experimental setup

Datasets: Skywork-OR1 math CodeMMLU-related code problems science reasoning problems Verifiable Instruction Following dataset AMC12 Math500 AIME 24/25 CodeMMLU GPQA-Diamond MMLU ZebraLogic MedQA

Benchmarks: AMC12 Math500 AIME 24/25 CodeMMLU GPQA-Diamond MMLU ZebraLogic MedQA Scruples WMDP sandbagging HumanEval+ MBPP+

Metrics: g-mean² monitorability draft-to-answer faithfulness (draft reliance, draft-answer c avg@n accuracy pearson correlation response entropy cross-segment attention mass

Baselines: Math-only Code-only Science-only IF-only All w/o IF

Builds on: GRPO DeepSeek-R1-Distill-Qwen-1.5B Qwen3-4B g-mean² monitorability metric draft-to-answer faithfulness

Closest papers by meaning

• Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model? (NeurIPS 2025 · similarity 0.90)
• Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMs (ICLR 2026 · similarity 0.90)
• Detecting RLVR Training Data via Structural Convergence of Reasoning (NeurIPS 2026 · similarity 0.90)
• The Reasoning Boundary Paradox: How Reinforcement Learning Constrains Language Models (NeurIPS 2026 · similarity 0.90)
• Outcome Rewards Do Not Guarantee Verifiable or Causally Important Reasoning (ICML 2026 · similarity 0.90)
• Exploration vs Exploitation: Rethinking RLVR through Clipping, Entropy, and Spurious Reward (ICLR 2026 · similarity 0.90)