Research map / Large language models / Reasoning and chain of thought

Exploration vs Exploitation: Rethinking RLVR through Clipping, Entropy, and Spurious Reward

ICLR 2026 · Large language models › Reasoning and chain of thought · approach: RL with verifiable rewards for reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Analyzes why random/spurious rewards improve RLVR, showing clipping bias gives no real learning signal but reduces policy entropy, and proposes a reward-misalignment model explaining when spurious rewards help across Qwen-Math, Llama, and QwQ models.

Problem

Spurious (random) rewards and entropy minimization both appear to improve RLVR reasoning, which seems paradoxical; it is unclear whether clipping bias, policy entropy, or contamination explains these gains.

Approach

Derives bounds showing upper-clipping correction is much smaller than the raw surrogate under random rewards, links clipping to a one-step entropy decrease, and introduces a reward-misalignment (false positive/negative) model predicting that stronger models benefit more from random rewards.

Key results

• Under random rewards on Qwen2.5-Math-7B, disabling clipping often improves MATH500 validation accuracy while enabling clipping tends to degrade it; clipping activation ratio never exceeds 0.2%.
• Theoretical corollary gives E|N_raw|/E|C_tot+| >= approximately 17.15, so clipping correction is negligible relative to the raw surrogate.
• Clipping under random rewards monotonically decreases policy entropy, while unclipped training can increase entropy, and entropy changes do not causally determine accuracy gains.

Surprising findings

• Entropy can increase while validation performance improves under unclipped random-reward training, so entropy is not causally tied to performance.
• Random rewards improve performance on Llama and QwQ families, not only the potentially contaminated Qwen-Math.

Limitations

• Theoretical analysis relies on a softmax tabular NPG surrogate and a clip-activation-rate abstraction of multi-step GRPO updates.
• Results across prior works are not directly comparable due to differing prompts and hyperparameters.
• Reward-misalignment model is a simple probabilistic warm-up and is only partially validated empirically.

Experimental setup

Datasets: DeepScaleR AIME Past

Benchmarks: MATH500 AIME

Metrics: validation accuracy policy entropy clipping activation ratio percentage improvement

Baselines: unclipped GRPO clipped GRPO

Builds on: GRPO PPO DeepScaleR verl

Closest papers by meaning

• Reasoning with Exploration: An Entropy Perspective (AAAI 2026 · similarity 0.93)
• Spurious Rewards: Rethinking Training Signals in RLVR (ICML 2026 · similarity 0.93)
• Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMs (ICLR 2026 · similarity 0.93)
• The Reasoning Boundary Paradox: How Reinforcement Learning Constrains Language Models (NeurIPS 2026 · similarity 0.92)
• Diversity-Incentivized Exploration for Versatile Reasoning (ICLR 2026 · similarity 0.92)
• Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning (NeurIPS 2025 · similarity 0.92)