Research map / Large language models / Reasoning and chain of thought

Do Not Let Low-Probability Tokens Over-Dominate in RL for LLMs

ICLR 2026 · Large language models › Reasoning and chain of thought · approach: RL with verifiable rewards for reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Identifies that low-probability tokens dominate GRPO gradient updates in LLM RL and proposes Advantage Reweighting and Low-Probability Token Isolation (Lopti) to attenuate their influence, improving GRPO on K&K logic puzzles and math tasks.

Problem

In RL post-training of LLMs with GRPO, low-probability tokens have disproportionately large gradients and dominate updates, suppressing learning on high-probability tokens.

Approach

Shows theoretically that the gradient norm w.r.t. activations for a token is bounded by a term proportional to (1 - pi), so low-probability tokens dominate; then down-weights advantages of low-probability tokens (Advantage Reweighting) or updates them in a separate stage before high-probability tokens (Lopti).

Key results

• On K&K puzzles with Qwen2.5-3B-Instruct, Advantage Reweighting and Lopti improve naive GRPO by 35.9% and 38.5% respectively, and by 46.2% when combined.
• On math with Qwen2.5-7B and DSR-Uniform, GRPO average accuracy rises from 38.98 to 40.01 with Reweight and 39.59 with Lopti.
• On ORZ with Qwen2.5-7B, GRPO average rises from 39.83 to 41.09 with Reweight and 40.66 with Lopti.

Surprising findings

• Reversing the Lopti update order (high-probability tokens first) makes training collapse after the 4th epoch, performing worse than GRPO.
• Combining Advantage Reweighting and Lopti does not help on math datasets, though it helps on K&K puzzles.

Limitations

• Lopti performs two update passes, increasing computational cost.
• Advantage Reweighting's alpha is task-sensitive, requiring careful tuning (0.3 for K&K, 0.1 for math).
• Gains on math benchmarks are modest and the method was tested on a limited set of base models.

Experimental setup

Datasets: K&K logic puzzles (Logic-RL) DeepScaleR-derived DSR-Uniform (AdaRFT subset) Open-Reasoner-Zero

Benchmarks: K&K logic puzzles OlympiadBench Minerva MATH-500 AMC AIME24

Metrics: accuracy avg@16 pass@16 reward

Baselines: GRPO DAPO clip-higher

Builds on: GRPO PPO DAPO Logic-RL

Closest papers by meaning

• Accelerating RL for LLM Reasoning with Optimal Advantage Regression (NeurIPS 2025 · similarity 0.92)
• STAPO: Stabilizing Reinforcement Learning for LLMs by Silencing Rare Spurious Tokens (NeurIPS 2026 · similarity 0.92)
• Diversity-Aware Policy Optimization for Large Language Model Reasoning (NeurIPS 2025 · similarity 0.92)
• Reshaping Reasoning in LLMs: A Theoretical Analysis of RL Training Dynamics through Pattern Selection (ICLR 2026 · similarity 0.92)
• On Predictability of Reinforcement Learning Dynamics for Large Language Models (ICLR 2026 · similarity 0.91)
• Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning (NeurIPS 2025 · similarity 0.91)