Research map / Large language models / Reasoning and chain of thought

Spurious Rewards: Rethinking Training Signals in RLVR

ICML 2026 · Large language models › Reasoning and chain of thought · approach: RL with verifiable rewards for reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows that RLVR with GRPO on Qwen2.5-Math models yields large MATH-500 and AMC gains even under spurious (random, incorrect, format) rewards, attributes this to GRPO's clipping bias amplifying high-prior code-reasoning behavior, and finds the effect does not transfer to Llama, OLMo2 or other families.

Problem

RLVR is widely believed to need accurate reward signals, but it is unclear whether gains in math reasoning reflect reward informativeness or pre-existing model behaviors, and whether conclusions from Qwen models generalize.

Approach

Replace ground-truth rewards with weak (majority vote, format) and spurious (random, majority-voted incorrect) binary rewards and show gains on Qwen2.5-Math; explain via a clipping bias in GRPO that asymmetrically upweights high-probability tokens, and link gains to amplified code-reasoning behavior.

Key results

• Random rewards improve Qwen2.5-Math-7B MATH-500 by 21.4 points vs. 29.1 points with ground-truth rewards.
• Code reasoning frequency in Qwen2.5-Math-7B rises from 65% to over 90% under spurious rewards; responses with code reach 60.9% accuracy vs 28.0% without (per abstract/text).
• Spurious rewards that help Qwen models yield minimal or negative gains on Llama3 and OLMo2 models; forcing Python prompting drops Llama3.1-8B accuracy by 21.6 points.

Surprising findings

• Random rewards yield substantial MATH-500 gains on Qwen2.5-Math models, and removing the GRPO clipping term eliminates this effect.
• Incorrect (majority-voted wrong-label) rewards nearly match ground-truth gains on Qwen2.5-Math-7B.

Limitations

• Spurious-reward findings are shown mainly on Qwen2.5-Math models and a limited set of other families.
• Clipping-bias explanation is partly conjectural and validated empirically rather than fully proven.
• Code reasoning is used as a case study and not an exhaustive account of Qwen behavior.

Experimental setup

Datasets: DeepScaleR MATH-500 AMC AIME 2024 AIME 2025

Benchmarks: MATH-500 AMC AIME 2024 AIME 2025

Metrics: pass@1 average@8 accuracy code reasoning frequency

Baselines: ground truth reward majority vote reward format reward random reward incorrect reward TTRL one-shot RL

Builds on: GRPO DeepSeekMath TTRL one-shot RL DeepScaleR

Closest papers by meaning

• Exploration vs Exploitation: Rethinking RLVR through Clipping, Entropy, and Spurious Reward (ICLR 2026 · similarity 0.93)
• Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMs (ICLR 2026 · similarity 0.93)
• Reinforcement Learning for Reasoning in Large Language Models with One Training Example (NeurIPS 2025 · similarity 0.93)
• Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model? (NeurIPS 2025 · similarity 0.92)
• From Verifiable Dot to Reward Chain: Harnessing Verifiable Reference-based Rewards for Reinforcement Learning of Open-ended Generation (ICLR 2026 · similarity 0.91)
• The Reasoning Boundary Paradox: How Reinforcement Learning Constrains Language Models (NeurIPS 2026 · similarity 0.91)