Research map / Reinforcement learning / RL from feedback

STAPO: Stabilizing Reinforcement Learning for LLMs by Silencing Rare Spurious Tokens

NeurIPS 2026 · Reinforcement learning › RL from feedback · approach: RL for LLM reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes STAPO, which masks the ~0.01% of low-probability, low-entropy tokens inside correct responses (S2T mechanism) during GRPO/DAPO-style RLVR to stabilize entropy and improve math reasoning across Qwen3 1.7B/8B/14B base models.

Problem

RL fine-tuning of LLMs for reasoning (RLVR) suffers from entropy explosion or collapse and late-stage performance collapse, which existing heuristic stabilizers (entropy regularization, reweighting, clipping changes) address only coarsely.

Approach

Identify "spurious tokens" — rare (low-probability) tokens inside correct responses that sit in low-entropy contexts — which receive large gradients under sequence-level rewards; mask their gradient contributions (S2T) with an entropy quantile threshold, rather than masking all low-probability tokens.

Key results

• STAPO at Qwen3-8B-Base reaches 60.96 average accuracy vs 57.63 for 20-Entropy and 56.24 for GRPO under the training-aligned setting (temp 1.0).
• At Qwen3-14B-Base, STAPO averages 67.76 vs 61.49 for GRPO (JustRL setting shows 67.80 vs 66.71 best baseline).
• Masked spurious tokens stay below 0.01% of tokens after warm-up; STAPO avoids the entropy explosion of JustRL and 20-Entropy and the collapse of GRPO.

Surprising findings

• Fully masking all low-probability tokens in correct responses (JustRL-FullMask) causes severe entropy collapse and degraded performance, so low-probability tokens are not uniformly harmful.
• Masking high-entropy low-probability tokens significantly degrades results, while masking low-entropy ones is best.

Limitations

• Analysis focuses only on tokens in correct responses that were rewarded; tokens in incorrect responses are not addressed.
• Evaluation limited to mathematical reasoning benchmarks.
• Relies on hyperparameters tau_p and q, with performance gaps narrowing under the JustRL decoding setting.

Experimental setup

Datasets: DAPO-Math-17K

Benchmarks: AIME24 AIME25 AMC23 MATH500 Minerva OlympiadBench

Metrics: average accuracy (avg@N) policy entropy training reward

Baselines: GRPO 20-Entropy JustRL

Builds on: GRPO DAPO JustRL 20-Entropy veRL

Closest papers by meaning

• Do Not Let Low-Probability Tokens Over-Dominate in RL for LLMs (ICLR 2026 · similarity 0.92)
• STARE: Surprisal-Guided Token-Level Advantage Reweighting for Policy Entropy Stability (NeurIPS 2026 · similarity 0.91)
• Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning (NeurIPS 2025 · similarity 0.90)
• One-Way Policy Optimization for Self-Evolving LLMs (ICML 2026 · similarity 0.90)
• Distributionally Robust Token Optimization in RLHF (NeurIPS 2026 · similarity 0.89)
• On Predictability of Reinforcement Learning Dynamics for Large Language Models (ICLR 2026 · similarity 0.89)