AI Paper Atlas

Research map / Reinforcement learning / RL from feedback

Reward Budgeting Reduces Premature Convergence in Reinforcement Learning for LLM Reasoning

Mengni Jia, Mengyu Zhou, xiaoxi jiang, Guanjun Jiang

NeurIPS 2026 · Reinforcement learning › RL from feedback · approach: Scaling RL with verifiable rewards · 0 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes controlling a sharpness factor κ, identified via analysis of RL dynamics through the optimal policy, to reduce premature plateaus in RL for LLM reasoning.

Problem

RL fine-tuning of LLMs for reasoning often plateaus early as policies sharpen quickly, with little further gain despite remaining model and data capacity.

Approach

Analyzes RL dynamics through the optimal policy to identify a sharpness factor κ as a better intervention target than entropy, which the authors argue is not sufficient on its own; a reward budgeting intervention is built around this factor.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
add entropy bonus to loss for first 500 steps, then revert to baseline (ablation)GRPO · Qwen3-8B on DAPO-Math-17Kfinal validation accuracyNo clear effect
add entropy to reward for first 500 steps, then revert to baseline (ablation)GRPO · Qwen3-8B on DAPO-Math-17Kfinal validation accuracyNo clear effect
add reward budgeting to GRPOGRPO · Qwen3-8B, five-benchmark averageaverage accuracy: 59.87 -> +6.66Improved
add reward budgeting to six RL baselinesGRPO, DrGRPO, GSPO, DAPO, ClipCov, KLCov · Qwen3-8B and Llama3.2-3B-Instructaverage score: up to +12.50 on AMC 23 and +10.00 on AIME24 & AIME25Improved
add reward budgeting to DrGRPO on MATH500DrGRPO · Qwen3-8B MATH500accuracy: -0.20No clear effect
use entropy-based variants ClipCov/KLCov instead of DAPODAPO · Qwen3-8B averageaverage accuracy: DAPO 61.94 vs ClipCov 57.98 vs KLCov 61.11Mixed
remove reference KL penalty from GRPO (ablation)GRPO · Qwen3-8B averageaverage accuracy: 59.87 -> 60.27No clear effect
add reward budgeting without KL (ablation)GRPO without KL · Qwen3-8B averageaverage accuracy: 60.27 -> 65.65Improved
reduce sharpness factor kappa from 5000 to 200 via reward scale and beta (ablation)DrGRPO · Qwen3-8B on DAPO-Math-17Kfinal performance and convergence timeMixed
apply reward budgetingQwen3-8B baselines · Qwen3-8B training entropytraining entropyImproved

Surprising findings

Experimental setup

Baselines: entropy promotion

Builds on: entropy regularization

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy