Research map / Large language models / Reasoning and chain of thought

The Reasoning Boundary Paradox: How Reinforcement Learning Constrains Language Models

NeurIPS 2026 · Large language models › Reasoning and chain of thought · approach: RL with verifiable rewards for reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Analyzes why RLVR shrinks pass@k coverage via negative interference and winner-take-all dynamics, and proposes SELF, a data curation method that trains only on problems where the greedy response fails, with a forward-KL term.

Problem

RLVR can improve pass@1 yet shrink the set of problems a model can solve within k attempts (reasoning boundary shrinkage); the cause of this coverage loss during RLVR training is unclear.

Approach

Uses per-step influence (NTK-style gradient kernel) to define negative interference and shows on-policy RL concentrates updates on high-likelihood correct solutions (winner-take-all); proposes SELF, which only trains on prompts where the greedy response fails and replaces reverse KL with a forward-KL SFT term.

Key results

• On Qwen2.5-Math-7B, SELF reaches Pass@1024 of 89.62 on AIME 2024 vs 84.77 for GRPO and 81.83 for the base model.
• On Llama-3.2-3B-Instruct, SELF raises Minerva Pass@256 to 57.35 vs 50.73 for GRPO and 49.26 for the base model.
• On Qwen2.5-Math-1.5B, SELF reaches Pass@1024 of 69.45 on AIME 2025 vs 64.05 for GRPO.

Surprising findings

• Larger model (Qwen2.5-Math-7B) suffers more negative interference and can underperform the smaller 1.5B model on large-k Pass@k after GRPO.
• Clipping violations stay stable on the training batch but grow on out-of-batch probing data, so clipping does not constrain interference.

Limitations

• Evaluated only on mathematical reasoning benchmarks and three models.
• SELF achieves lower average training reward than GRPO and sometimes lower Pass@1 (e.g., AIME 2024 on 7B).
• Analysis relies on a first-order approximation and a toy bandit, with the learning-dynamics theory only loosely tied to LLM training.

Experimental setup

Datasets: DeepScaleR Math500 AIME 2024 AIME 2025 Minerva

Benchmarks: AIME 2024 AIME 2025 Math500 Minerva

Metrics: Pass@1 Pass@k (k=256 or 1024) token-level entropy perplexity

Baselines: GRPO Base model

Builds on: GRPO PPO VERL DeepScaleR

Closest papers by meaning

• Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMs (ICLR 2026 · similarity 0.94)
• Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model? (NeurIPS 2025 · similarity 0.94)
• The Unlearnability Phenomenon in RLVR for Language Models (ICML 2026 · similarity 0.93)
• Reinforcement Learning for Reasoning in Large Language Models with One Training Example (NeurIPS 2025 · similarity 0.93)
• The Surprising Effectiveness of Negative Reinforcement in LLM Reasoning (NeurIPS 2025 · similarity 0.93)
• On the Direction of RLVR Updates for LLM Reasoning: Identification and Exploitation (ICLR 2026 · similarity 0.93)