Research map / Large language models / Reasoning and chain of thought

Outcome Rewards Do Not Guarantee Verifiable or Causally Important Reasoning

ICML 2026 · Large language models › Reasoning and chain of thought · approach: RL with verifiable rewards for reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces Causal Importance of Reasoning (CIR) and Sufficiency of Reasoning (SR) metrics and shows outcome-only RLVR does not reliably improve them, while small SFT before RLVR or auxiliary CIR/SR rewards do, without hurting accuracy.

Problem

Whether chain-of-thought reasoning traces trained via outcome-based RLVR are causally important for the final answer and verifiable (sufficient for an external verifier to reach the same answer).

Approach

Two metrics: CIR measures JS divergence between answer distributions under progressively truncated reasoning vs full reasoning; SR measures whether a strong verifier gives the same answer from sanitized reasoning with and without the question. Then uses small expert-trace SFT before RLVR, or adds CIR/SR as auxiliary rewards to the outcome reward.

Key results

• Across 40 ReasoningGym tasks with Qwen2.5-3B, 19 tasks show CIR decline and 17 show SR decline after RLVR, despite accuracy gains.
• SFT on 64 or 512 expert traces before RLVR raises final CIR to about 0.4 and SR to about 0.65-0.75 on low-CIR/SR algorithmic tasks.
• Adding CIR or SR as auxiliary reward (alpha=beta=1.0) keeps accuracy comparable to outcome-only RLVR while improving CIR/SR; on MATH Level 5 CIR rose 0.23 to 0.51 and accuracy 0.11 to 0.56.

Surprising findings

• For tasks where final CIR and SR are near zero, training to directly answer without reasoning yields similar accuracy gains to reasoning-based RLVR.
• Models often reach a final answer before emitting the think segment, so truncating within think leaves the prediction essentially unchanged.

Limitations

• SR is approximated via greedy verifier agreement and is permissive; reliance on gpt-4o-mini and API costs limited experiments.
• Auxiliary CIR/SR rewards do not improve accuracy on tasks where base accuracy is near zero, and CIR reward uses a coarse 10-segment approximation.
• Experiments limited to ReasoningGym tasks and small Qwen/Llama models with a 1024-token thinking budget.

Experimental setup

Datasets: ReasoningGym (40 tasks) MATH Level 5 (Math-Hard)

Benchmarks: ReasoningGym (40 tasks) MATH Level 5

Metrics: accuracy CIR SR SR-minus (stricter sufficiency)

Baselines: outcome-only RLVR direct-answer RLVR SL-before-RL

Builds on: GRPO ReasoningGym Qwen2.5

Closest papers by meaning

• Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMs (ICLR 2026 · similarity 0.94)
• The Reasoning Boundary Paradox: How Reinforcement Learning Constrains Language Models (NeurIPS 2026 · similarity 0.93)
• From Verifiable Dot to Reward Chain: Harnessing Verifiable Reference-based Rewards for Reinforcement Learning of Open-ended Generation (ICLR 2026 · similarity 0.92)
• Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model? (NeurIPS 2025 · similarity 0.92)
• Spurious Rewards: Rethinking Training Signals in RLVR (ICML 2026 · similarity 0.91)
• A Simple "Motivation" Can Enhance Reinforcement Finetuning of Large Reasoning Models (ICLR 2026 · similarity 0.91)