Research map / Large language models / Reasoning and chain of thought

What are Key Factors for Updates in RL for LLM Reasoning?

NeurIPS 2026 · Large language models › Reasoning and chain of thought · approach: RL with verifiable rewards for reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Analyzes how off-policy degree (updates per rollout) reshapes clipped GRPO/RLVR gradients via IS-ratio variance, and proposes ACPO, which sets per-probability-bin clip ranges from within-bin IS-ratio std.

Problem

Heuristic token-level RLVR choices (high-entropy vs. low-probability token emphasis) report contradictory results; the paper seeks to explain why, in terms of off-policy degree and clipping in GRPO-style updates.

Approach

Off-policy degree widens IS-ratio dispersion, especially for low-probability tokens, so clipping shifts which token group dominates the expected gradient (a "gradient dominance reversal"). Based on this, ACPO replaces a global clip range with bin-wise clip ranges adapted to the within-bin IS-ratio standard deviation.

Key results

• ACPO achieves average 39.86 on six math benchmarks with Qwen2.5-7B on ORZ-57K under near on-policy training, vs. 38.70 for DAPO and 37.93 for CISPO.
• On HiTab with Qwen2.5-7B, ACPO reaches 69.83 (N-OnP.) and 66.58 (OffP.), the best in both regimes.
• ACPO achieves average rank 1.22 across 18 task-regime configurations, with 15 first-place finishes; ACPO drops only 0.8% from near on-policy to off-policy, versus 6.0% for CISPO.

Surprising findings

• CISPO and AR-Lopti degrade substantially under off-policy training (-6.0% and -5.0%), while High/Low-Entropy methods are stable but lower overall.
• Clipping alone rotates the update direction by about 47 degrees relative to the on-policy gradient despite similar gradient norms.

Limitations

• The closed-form gradient expectation relies on modeling assumptions (log-normal IS ratio, linear advantage coupling, perfect-coupling idealization) and the finite-noise case is not developed.
• Experiments are limited to Qwen2.5 models at 3B and 7B and three task suites.

Experimental setup

Datasets: ORZ-57K HiTab Countdown

Benchmarks: Minerva MATH500 AMC AIME24 AIME25 OlympiadBench HiTab Countdown

Metrics: accuracy avg@32 avg@8 avg@4

Baselines: DAPO CISPO High-Entropy token update Low-Entropy token update AR-Lopti

Builds on: GRPO DAPO PPO

Closest papers by meaning

• On the Direction of RLVR Updates for LLM Reasoning: Identification and Exploitation (ICLR 2026 · similarity 0.94)
• Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMs (ICLR 2026 · similarity 0.92)
• The Reasoning Boundary Paradox: How Reinforcement Learning Constrains Language Models (NeurIPS 2026 · similarity 0.92)
• Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model? (NeurIPS 2025 · similarity 0.91)
• Do Not Let Low-Probability Tokens Over-Dominate in RL for LLMs (ICLR 2026 · similarity 0.91)
• Depth-Breadth Synergy in RLVR: Unlocking LLM Reasoning Gains with Adaptive Exploration (ICML 2026 · similarity 0.91)