Research map / Reinforcement learning / RL from feedback

SCOPE-RL: Stable and Quantitative Control of Policy Entropy in RL Post-Training

NeurIPS 2026 · Reinforcement learning › RL from feedback · approach: RL for LLM reasoning · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes SCOPE-RL, which adds a small temperature-adaptive positive-sample regularizer to GRPO to keep policy entropy in a target range and avoid entropy collapse in RL post-training of LLMs.

Problem

GRPO-based RL post-training of reasoning LLMs suffers from entropy collapse, and existing fixes (entropy bonuses, clip-higher) produce unstable entropy oscillations and reward degradation.

Approach

Under high-temperature sampling, positive samples raise entropy while negative samples lower it; a theorem in a simplified bandit setting shows that the entropy drift's temperature derivative is positive under positive-sample updates when entropy is decreasing. SCOPE-RL uses a small batch of positive samples drawn at a temperature adapted to the gap between current entropy and a target H0 as an au

Key results

• On Qwen2.5-Math-7B, SCOPE-RL averages 54.84 vs 51.60 for GRPO (+3.23) across seven math benchmarks.
• On Qwen2.5-7B, SCOPE-RL reaches 86.7 Pass@1024 on AIME24 vs 73.3 for GRPO, and 76.7 vs 63.3 on AIME25.
• On Qwen3-4B, average math score rises from 67.77 (GRPO) to 69.73 (SCOPE-RL).

Surprising findings

• Negative samples drawn at high temperature accelerate entropy collapse, while positive samples at high temperature raise entropy, reversing under low temperature.
• Larger auxiliary sample ratio (128/4096) increases entropy but causes larger reward fluctuations, so benefit is non-monotonic in exploration level.

Limitations

• Theory relies on a tabular softmax bandit with orthogonal logit gradients and a first-order approximation, far from full autoregressive LLM training.
• Theorem does not cover regimes where explicit entropy control makes entropy drift positive.
• Entropy control is global; state-dependent exploration is left for future work.

Experimental setup

Datasets: DAPO-17K

Benchmarks: AIME24 AIME25 AMC23 GSM8K MATH Minerva Math OlympiadBench ARC-Challenge MMLU-Pro SuperGPQA

Metrics: Pass@1 Pass@k (k=512, 1024) average accuracy

Baselines: GRPO Entropy-Reg Entropy-Adv KL-cov DAPO CISPO Clip higher

Builds on: GRPO DAPO Entropy regularization KL-Cov CISPO

Closest papers by meaning

• STARE: Surprisal-Guided Token-Level Advantage Reweighting for Policy Entropy Stability (NeurIPS 2026 · similarity 0.90)
• Reasoning with Exploration: An Entropy Perspective (AAAI 2026 · similarity 0.89)
• Exploration vs Exploitation: Rethinking RLVR through Clipping, Entropy, and Spurious Reward (ICLR 2026 · similarity 0.89)
• Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning (NeurIPS 2025 · similarity 0.88)
• Reshaping Reasoning in LLMs: A Theoretical Analysis of RL Training Dynamics through Pattern Selection (ICLR 2026 · similarity 0.88)
• Trajectory Balance with Asynchrony: Decoupling Exploration and Learning for Fast, Scalable LLM Post-Training (NeurIPS 2025 · similarity 0.88)