Research map / Reinforcement learning / RL from feedback
Token Hidden Reward: Steering Exploration-Exploitation in Group Relative Deep Reinforcement Learning
Wenlong Deng, Yi Ren, Yushu Li, Boying Gong, Danica J. Sutherland, Xiaoxiao Li, Christos Thrampoulidis
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| train only on dominant (high |THR|) tokens (ablation) | GRPO · math benchmarks, greedy total avg | accuracy (%): 42.7 -> 43.3 | No clear effect |
| amplify positive-THR tokens (p=0.1) (ablation) | THR · math benchmarks greedy total avg | accuracy (%): 43.3 -> 46.8 | Improved |
| amplify negative-THR tokens (p<0) (ablation) | THR · Pass@256 average, AIME/AMC | Pass@256 (%): 72.5 -> 76.7 | Improved |
| amplify negative-THR tokens (p<0) vs positive on greedy accuracy (ablation) | THR · math benchmarks greedy total avg | accuracy (%): Qwen2.5-Math-7B: 46.8 (p>0) vs 44.0 (p<0); 1.5B: 36.3 vs 36.4 | Mixed |
| use THR (p<0) instead of Pass@K-mixed training | Qwen2.5-Math-1.5B · Pass@K average | Pass@256 (%): 73.3 -> 76.7 | Improved |
| combine THR (p<0) directly with Pass@K-mixed (ablation) | THR (p<0) · Pass@K average | Pass@1 (%): 29.7 -> 27.3 (Qwen2.5-Math-7B) | Hurt |
| combine THR (p<0) with static chi=0.2 Pass@K mix (ablation) | THR (p<0) · Pass@K average | Pass@1 (%): 29.7 -> 30.2 (Qwen2.5-Math-7B) | Improved |
| use THR (p<0) instead of Cov-KL entropy regularization | Qwen2.5-Math-1.5B · Pass@K average | Pass@256 (%): 69.2 -> 76.7 | Improved |
| add THR (p=-0.1) to GSPO | GSPO · Pass@K average | Pass@256 (%): 69.5 -> 73.3 | Improved |
| THR (p<0) vs GRPO on 7B at large K on AIME24 | Qwen2.5-Math-7B · AIME 2024 | Pass@256 (%): 76.7 -> 73.3 | Hurt |
Closest papers by meaning
- DeepSearch: Overcome the Bottleneck of Reinforcement Learning with Verifiable Rewards via Monte Carlo Tree Search (ICLR 2026 · similarity 0.89)
- Cornerstones or Stumbling Blocks? Deciphering the Rock Tokens in On-Policy Distillation (NeurIPS 2026 · similarity 0.88)
- Exploration vs Exploitation: Rethinking RLVR through Clipping, Entropy, and Spurious Reward (ICLR 2026 · similarity 0.88)
- STAPO: Stabilizing Reinforcement Learning for LLMs by Silencing Rare Spurious Tokens (NeurIPS 2026 · similarity 0.88)
- On the Effect of Negative Gradient in Group Relative Deep Reinforcement Optimization (NeurIPS 2025 · similarity 0.88)
- STARE: Surprisal-Guided Token-Level Advantage Reweighting for Policy Entropy Stability (NeurIPS 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
