Research map / Reinforcement learning / RL from feedback

STARE: Surprisal-Guided Token-Level Advantage Reweighting for Policy Entropy Stability

NeurIPS 2026 · Reinforcement learning › RL from feedback · approach: GRPO stability and variants · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes STARE, which analyzes token-level entropy dynamics under GRPO and reweights high-surprisal tokens in positive-advantage rollouts, gated by a target-entropy closed loop, to stabilize policy entropy during RLVR across 1.5B–32B models.

Problem

GRPO-style RLVR suffers policy entropy collapse during long training, and trajectory-level advantages fail to distinguish the opposing entropy effects of individual tokens.

Approach

A first-order gradient analysis gives a per-token entropy change equal to the advantage times an entropy sensitivity function, yielding an advantage-surprisal four-quadrant structure; STARE upweights positive-advantage high-surprisal tokens (selected by batch-level surprisal quantiles) and gates this by whether batch entropy falls below a target.

Key results

• STARE-O1 reaches 54.4% average on six math benchmarks at 7B Short CoT vs 49.1% for GRPO-ds, with AIME24 44.2 and AIME25 23.8.
• On DeepSeek-R1-Distill-Qwen-1.5B (Long CoT), STARE-O1 averages 65.9% vs GRPO-ds 62.5% and DAPO 55.1%.
• In the tool-use scenario on Qwen2.5-7B-Base, STARE-O1 averages 59.4% vs GRPO-ds 53.9%.

Surprising findings

• Open-loop reweighting with W=1.01 already mitigates entropy decay, while W≥2.0 triggers divergence.
• Open-loop reweighting stabilizes entropy at an excessively high level, causing over-exploration, which the target-entropy gate avoids.

Limitations

• Theoretical near-criticality result relies on first-order analysis and statistical assumptions about large T and batch size.
• Hyperparameters (W, M, P, H_tgt) are tuned on the same benchmarks reported; target entropy value sensitive (0.1–0.4 ablation).

Experimental setup

Datasets: DeepScaler Skywork-o1 Polaris DAPO Retool 2K

Benchmarks: AIME24 AIME25 AMC23 MATH-500 Minerva Math OlympiadBench

Metrics: average accuracy (avg@N) Pass@32 policy entropy

Baselines: GRPO-ds DAPO 80/20 Rule EntroAdv STEER KL-Cov EAPO JustRL STAPO SimpleTIR

Builds on: GRPO DAPO KL-Cov

Closest papers by meaning

• STAPO: Stabilizing Reinforcement Learning for LLMs by Silencing Rare Spurious Tokens (NeurIPS 2026 · similarity 0.91)
• Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning (NeurIPS 2025 · similarity 0.91)
• Reasoning with Exploration: An Entropy Perspective (AAAI 2026 · similarity 0.90)
• SCOPE-RL: Stable and Quantitative Control of Policy Entropy in RL Post-Training (NeurIPS 2026 · similarity 0.90)
• Conditional Advantage Estimation for Reinforcement Learning in Large Reasoning Models (ICLR 2026 · similarity 0.89)
• ReLaX: Reasoning with Latent Exploration for Large Reasoning Models (CVPR 2026 · similarity 0.88)