Research map / Reinforcement learning / RL from feedback
Reward Budgeting Reduces Premature Convergence in Reinforcement Learning for LLM Reasoning
Mengni Jia, Mengyu Zhou, xiaoxi jiang, Guanjun Jiang
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes controlling a sharpness factor κ, identified via analysis of RL dynamics through the optimal policy, to reduce premature plateaus in RL for LLM reasoning.
Problem
RL fine-tuning of LLMs for reasoning often plateaus early as policies sharpen quickly, with little further gain despite remaining model and data capacity.
Approach
Analyzes RL dynamics through the optimal policy to identify a sharpness factor κ as a better intervention target than entropy, which the authors argue is not sufficient on its own; a reward budgeting intervention is built around this factor.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add entropy bonus to loss for first 500 steps, then revert to baseline (ablation) | GRPO · Qwen3-8B on DAPO-Math-17K | final validation accuracy | No clear effect |
| add entropy to reward for first 500 steps, then revert to baseline (ablation) | GRPO · Qwen3-8B on DAPO-Math-17K | final validation accuracy | No clear effect |
| add reward budgeting to GRPO | GRPO · Qwen3-8B, five-benchmark average | average accuracy: 59.87 -> +6.66 | Improved |
| add reward budgeting to six RL baselines | GRPO, DrGRPO, GSPO, DAPO, ClipCov, KLCov · Qwen3-8B and Llama3.2-3B-Instruct | average score: up to +12.50 on AMC 23 and +10.00 on AIME24 & AIME25 | Improved |
| add reward budgeting to DrGRPO on MATH500 | DrGRPO · Qwen3-8B MATH500 | accuracy: -0.20 | No clear effect |
| use entropy-based variants ClipCov/KLCov instead of DAPO | DAPO · Qwen3-8B average | average accuracy: DAPO 61.94 vs ClipCov 57.98 vs KLCov 61.11 | Mixed |
| remove reference KL penalty from GRPO (ablation) | GRPO · Qwen3-8B average | average accuracy: 59.87 -> 60.27 | No clear effect |
| add reward budgeting without KL (ablation) | GRPO without KL · Qwen3-8B average | average accuracy: 60.27 -> 65.65 | Improved |
| reduce sharpness factor kappa from 5000 to 200 via reward scale and beta (ablation) | DrGRPO · Qwen3-8B on DAPO-Math-17K | final performance and convergence time | Mixed |
| apply reward budgeting | Qwen3-8B baselines · Qwen3-8B training entropy | training entropy | Improved |
Surprising findings
- Entropy promotion alone is not a sufficient intervention target in the settings studied.
Experimental setup
Baselines: entropy promotion
Builds on: entropy regularization
Closest papers by meaning
- Exploration vs Exploitation: Rethinking RLVR through Clipping, Entropy, and Spurious Reward (ICLR 2026 · similarity 0.90)
- SCOPE-RL: Stable and Quantitative Control of Policy Entropy in RL Post-Training (NeurIPS 2026 · similarity 0.90)
- Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning (NeurIPS 2025 · similarity 0.89)
- Policy Filtration for RLHF to Mitigate Noise in Reward Models (ICML 2025 · similarity 0.89)
- SFT-then-RL Outperforms Mixed-Policy Methods for LLM Reasoning (NeurIPS 2026 · similarity 0.89)
- STARE: Surprisal-Guided Token-Level Advantage Reweighting for Policy Entropy Stability (NeurIPS 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
