Research map / Large language models / Alignment and preferences
On the Effect of Negative Gradient in Group Relative Deep Reinforcement Optimization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Identifies Lazy Likelihood Displacement (LLD) in GRPO, where negative gradients on incorrect responses shrink or reverse likelihood gains of correct ones, and proposes NTHR, which softly down-weights penalties on incorrect-response tokens whose hidden-embedding influence on correct responses is high.
Problem
GRPO's negative gradients on incorrect responses can cause the likelihood of correct responses to increase marginally or decrease (Lazy Likelihood Displacement), hurting math reasoning performance.
Approach
Extends DPO's likelihood displacement analysis to GRPO's group structure, showing that the negative-gradient impact on correct responses is governed by hidden-embedding similarity; NTHR uses correct responses as anchors to score each incorrect token and attenuates the penalty on high-scoring tokens (selective token penalization) instead of discarding samples.
Key results
- On Qwen2.5-Math-1.5B, NTHR reaches 41.94 average accuracy versus 41.14 for GRPO across five math benchmarks.
- On Qwen2.5-1.5B-Ins, NTHR averages 28.48 versus 26.96 for GRPO; on Qwen2.5-0.5B-Ins, 12.66 versus 11.72.
- On Qwen2.5-3B, NTHR averages 36.30 versus 33.88 for GRPO and 31.36 for the base model.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use NTHR selective token penalization instead of standard GRPO | Qwen2.5-Math-1.5B · MATH (levels 3–5), math benchmarks average | Avg.: 41.14 -> 41.94 | Helped |
| Use NTHR selective token penalization instead of standard GRPO | Qwen2.5-0.5B-Ins · MATH (levels 3–5), math benchmarks average | Avg.: 11.72 -> 12.66 | Helped |
| Use NTHR selective token penalization instead of standard GRPO | Qwen2.5-1.5B-Ins · MATH (levels 3–5), math benchmarks average | Avg.: 26.96 -> 28.48 | Helped |
| Use NTHR selective token penalization instead of standard GRPO | Qwen2.5-3B · MATH (levels 3–5), math benchmarks average | Avg.: 33.88 -> 36.30 | Helped |
| Use NTHR selective token penalization instead of standard GRPO | Qwen2.5-Math-1.5B · DeepScaler, math benchmarks average | Avg.: 37.80 -> 39.60 | Helped |
| Use NTHR selective token penalization instead of standard GRPO | Deepseek-1.5B · AIME24, AMC, MATH500, Minerva, Olympiad average | Avg.: 49.1 -> 50.0 | Helped |
| Mask negative advantages entirely (Pos Only) rather than use GRPO | Qwen2.5-Math-1.5B · MATH (levels 3–5), math benchmarks average | Avg.: 41.14 -> 39.84 | Hurt |
| Apply η to all incorrect tokens rather than selectively thresholding tokens (β = -∞) (ablation) | NTHR on Qwen2.5-Math-1.5B · MATH (levels 3–5), math benchmarks average | Avg.: 41.14 -> 40.00 | Hurt |
| Set threshold scale β to 0 (ablation) | NTHR on Qwen2.5-Math-1.5B · MATH (levels 3–5), math benchmarks average | Avg.: 41.14 -> 41.54 | Helped |
| Set threshold scale β to 0.1 (ablation) | NTHR on Qwen2.5-Math-1.5B · MATH (levels 3–5), math benchmarks average | Avg.: 41.14 -> 41.50 | Helped |
| Set threshold scale β to 1.0 (ablation) | NTHR on Qwen2.5-Math-1.5B · MATH (levels 3–5), math benchmarks average | Avg.: 41.14 -> 41.94 | Helped |
| Set penalty scale η to 1-p instead of 2·|0.5-p| (ablation) | NTHR on Qwen2.5-Math-1.5B and Qwen2.5-3B · MATH (levels 3–5), math benchmarks average | Avg.: 41.94 -> 42.66; 36.30 -> 34.68 | Mixed |
Surprising findings
- Pos Only (removing negative gradients entirely) yields a 1.3% average drop versus GRPO on Qwen2.5-Math-1.5B, so negative gradients are not inherently harmful.
- Masking all negative-response tokens (beta = -infinity ablation) underperforms GRPO (40.00 vs 41.14).
Limitations
- Improvements are small (often under 2 points average) and based on single runs with five small benchmarks where differences are a few problems.
- DeepSeek-1.5B experiments were limited to a 4k context, causing response length to decline and unstable results.
- Theory relies on unconstrained-features and gradient-flow approximations and drops clipping and KL terms.
Experimental setup
Datasets: MATH DeepScaleR AIME
Benchmarks: AIME24 AMC MATH500 Minerva Math OlympiadBench
Metrics: accuracy
Baselines: GRPO Pos Only GRPO+Random
Builds on: GRPO DPO DAPO
Closest papers by meaning
- As Simple as Fine-tuning: LLM Alignment via Bidirectional Negative Feedback Loss (ICLR 2025 · similarity 0.90)
- TIC-GRPO: Provable and Efficient Optimization for Reinforcement Learning from Human Feedback (NeurIPS 2026 · similarity 0.88)
- Earlier Tokens Contribute More: Learning Direct Preference Optimization From Temporal Decay Perspective (ICLR 2025 · similarity 0.88)
- $f$-GRPO & Beyond: Divergence-Based Reinforcement Learning Algorithms for General LLM Alignment (NeurIPS 2026 · similarity 0.88)
- Zeroth-Order Policy Gradient for Reinforcement Learning from Human Feedback without Reward Inference (ICLR 2025 · similarity 0.88)
- The Surprising Effectiveness of Negative Reinforcement in LLM Reasoning (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.