Research map / Large language models / Reasoning and chain of thought

The Surprising Effectiveness of Negative Reinforcement in LLM Reasoning

NeurIPS 2025 · Large language models › Reasoning and chain of thought · approach: RL with verifiable rewards for reasoning · 2 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Decomposes RLVR into positive-sample (PSR) and negative-sample (NSR) reinforcement, shows NSR-only training improves Pass@k across the spectrum, and proposes W-REINFORCE which downweights the positive term (lambda=0.1).

Problem

Understanding how RLVR uses correct versus incorrect samples and why it affects inference-scaling (Pass@k) behavior, including loss of output diversity when training with RL on reasoning tasks.

Approach

Splits the binary-reward RLVR objective into PSR (reinforce correct samples) and NSR (penalize incorrect samples); shows NSR alone preserves entropy and improves Pass@k, explained by a token-level gradient that redistributes mass by the model's prior; then proposes W-REINFORCE, which downweights the PSR term by lambda=0.1.

Key results

• NSR-only training on Qwen2.5-Math-7B reaches Pass@256 of 96.9 on MATH (matching base) and 53.3 on AIME 2025 vs 46.7 for base, while PSR falls below base for k>8.
• W-REINFORCE (lambda=0.1) achieves 76.6 Pass@1 on MATH, tying the best PPO score, and 56.7 Pass@256 on AIME 2025 vs 50.0 for GRPO.
• On Qwen3-4B (non-thinking), NSR reaches 94.0 Pass@1 and 98.0 Pass@64 on MATH, close to thinking mode (94.5 / 97.8).

Measured effects reported in the paper

Surprising findings

• Training only on negative samples improves Pass@1 nearly as much as PPO/GRPO on Qwen2.5-Math-7B, despite never reinforcing correct responses.
• PSR-only training fails to transfer Qwen3-4B's latent thinking-mode reasoning to non-thinking mode, while NSR does.

Limitations

• Extended NSR training (hundreds of steps) leads to performance decline, so some PSR may be needed.
• Study restricted to sparse binary outcome rewards; dense reward settings are untested.
• Experiments limited to math benchmarks and three base models.

Experimental setup

Datasets: MATH

Benchmarks: MATH AIME 2025 AMC23

Metrics: Pass@k entropy greedy accuracy

Baselines: PPO GRPO REINFORCE PSR base model unlikelihood training

Builds on: REINFORCE PPO GRPO RLVR unlikelihood training

Closest papers by meaning

• Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMs (ICLR 2026 · similarity 0.94)
• The Reasoning Boundary Paradox: How Reinforcement Learning Constrains Language Models (NeurIPS 2026 · similarity 0.93)
• Reinforcement Learning for Reasoning in Large Language Models with One Training Example (NeurIPS 2025 · similarity 0.93)
• Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model? (NeurIPS 2025 · similarity 0.93)
• On the Direction of RLVR Updates for LLM Reasoning: Identification and Exploitation (ICLR 2026 · similarity 0.92)
• ResRL: Boosting LLM Reasoning via Negative Sample Projection Residual Reinforcement Learning (ICML 2026 · similarity 0.92)