Research map / Large language models / Reasoning and chain of thought

On the Direction of RLVR Updates for LLM Reasoning: Identification and Exploitation

ICLR 2026 · Large language models › Reasoning and chain of thought · approach: RL with verifiable rewards for reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes signed token-level log-probability difference (Δlog p between RLVR and base models) as a directional diagnostic of RLVR updates, and uses it for test-time extrapolation and advantage reweighting toward low-probability tokens.

Problem

Prior analyses of RLVR-induced changes focus on the magnitude of token-level distribution shifts (entropy, KL), overlooking their direction, and it is unclear which sparse tokens drive reasoning gains.

Approach

Use the signed log-probability difference between RLVR and base models to identify reasoning-critical tokens; extrapolate the RLVR policy along this direction at selected positions at test time, and upweight advantages of low-probability tokens during training.

Key results

• Test-time extrapolation with selective gating raises DAPO-32B AIME-24 Avg@32 from 52.60 (pi_RL) to 55.42 (gamma=0.05, tau=-0.3), with similar gains on ORZ-32B and UniReason-14B.
• Reweighting advantages by (1 - pi_old) on Qwen2.5-Math-7B improves average Avg@32 over DAPO from 42.12 to 43.75 across AIME24, AIME25, and AMC.
• On Qwen3-8B-Base the reweighting raises average Avg@32 from 44.26 (DAPO) to 46.78 and average Pass@16 from 69.19 to 72.52.

Surprising findings

• Replacing only about 10% of base-model tokens selected by Δlog p recovers RLVR-level accuracy, fewer than entropy or divergence criteria need.
• Restricting rollouts to high-probability tokens via top-p filtering (even top-p=0.95) substantially degrades RLVR training performance.

Limitations

• Test-time extrapolation requires access to both the base and RLVR models.
• Extrapolation adds hyperparameters (threshold tau and strength gamma) that must be tuned.
• Experiments are limited to math-focused reasoning on a few model pairs and benchmarks; the reweighting is evaluated on only two base models.

Experimental setup

Datasets: AIME-24 AIME-25 AMC Minerva

Benchmarks: AIME-24 AIME-25 AMC Minerva

Metrics: Avg@32 Pass@16 Avg@8

Baselines: DAPO Base model PPL reweighting Dominate reweighting Selective Replace Entropy-based selection KL divergence-based selection

Builds on: GRPO DAPO PPO Open-Reasoner-Zero Natural Policy Gradient

Closest papers by meaning

• Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMs (ICLR 2026 · similarity 0.95)
• Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model? (NeurIPS 2025 · similarity 0.94)
• What are Key Factors for Updates in RL for LLM Reasoning? (NeurIPS 2026 · similarity 0.94)
• The Reasoning Boundary Paradox: How Reinforcement Learning Constrains Language Models (NeurIPS 2026 · similarity 0.93)
• Reinforcement Learning for Reasoning in Large Language Models with One Training Example (NeurIPS 2025 · similarity 0.93)
• The Unlearnability Phenomenon in RLVR for Language Models (ICML 2026 · similarity 0.92)