Research map / Reinforcement learning / RL from feedback
Demystifying Entropy Control in LLM RL Training: Theoretical Analysis and Dynamic Scheduling
Jingchu Gai, Guanning Zeng, Huaqing Zhang, Han Zhong, Yige Hong, Andrej Risteski, Aditi Raghunathan
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add dynamic entropy scheduling scaled by Entropy Discrepancy to GRPO | GRPO · Knights-and-Knaves KNK5, Pass@1 | Pass@1 gain (percentage points): +17.52 percentage points | Improved |
| add dynamic entropy scheduling scaled by Entropy Discrepancy to GRPO | GRPO · Math reasoning benchmarks (AIME24 etc.) | Pass@K gain (percentage points): up to +6.7 percentage points | Improved |
| add positive entropy regularization (coef 0.05) when Entropy Discrepancy is high (ablation) | GRPO on Countdown · Countdown, 39 tasks | Spearman correlation of gain vs discrepancy: Spearman 0.8039 | Improved |
| add negative entropy regularization (coef -0.01) when Entropy Discrepancy is low (ablation) | GRPO on Countdown · Countdown, 22 tasks | Spearman correlation of gain vs discrepancy: Spearman -0.8729 | Mixed |
| lower positive entropy coefficient to 0.01 (ablation) | GRPO on Countdown · Countdown, 28 tasks | Spearman correlation of gain vs discrepancy: Spearman 0.8749 | No clear effect |
| vary sampling temperature among 0.4, 0.7, 1.0 (ablation) | Dynamic entropy scheduling · Knights-and-Knaves | accuracy vs vanilla GRPO | Improved |
| apply an identical KL constraint at two coefficients to both methods (ablation) | Dynamic entropy scheduling · Knights-and-Knaves | accuracy gain over GRPO: around 10 percentage points on average | Improved |
| use dynamic schedule instead of grid-searched fixed entropy coefficient (ablation) | Entropy regularization · Knights-and-Knaves and math reasoning | accuracy | No clear effect |
| use dynamic entropy schedule instead of PKPO, Reward Unlikely, CISPO | RL reasoning methods · math reasoning | Pass@K / accuracy | No clear effect |
| static entropy increase vs decrease depends on task (ablation) | GRPO · Knights-and-Knaves vs math reasoning | Pass@1 / Pass@K | Mixed |
Closest papers by meaning
- SCOPE-RL: Stable and Quantitative Control of Policy Entropy in RL Post-Training (NeurIPS 2026 · similarity 0.93)
- Reasoning with Exploration: An Entropy Perspective (AAAI 2026 · similarity 0.91)
- AEM: Adaptive Entropy Modulation for Multi-Turn Agentic Reinforcement Learning (NeurIPS 2026 · similarity 0.91)
- The Unreasonable Effectiveness of Entropy Minimization in LLM Reasoning (NeurIPS 2025 · similarity 0.91)
- Exploration vs Exploitation: Rethinking RLVR through Clipping, Entropy, and Spurious Reward (ICLR 2026 · similarity 0.90)
- Reshaping Reasoning in LLMs: A Theoretical Analysis of RL Training Dynamics through Pattern Selection (ICLR 2026 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
