Research map / Reinforcement learning / RL from feedback
Tackling Length Inflation Without Trade-offs: Group Relative Reward Rescaling for Reinforcement Learning
Zichao Li, Jie Lou, Fangchen Dong, Zhiyuan Fan, Mengjie Ren, Hongyu Lin, Xianpei Han, Debing Zhang, Le Sun, Yaojie Lu, XingYu Li
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use GR3 multiplicative reward rescaling instead of GRPO | GRPO / GR3 · AIME24, R1-Distill-7B | Avg@32: 57.1 -> 60.1 | Improved |
| use GR3 instead of GRPO, tokens | GRPO / GR3 · AIME24, R1-Distill-7B | tokens: 11079 -> 7923 | Improved |
| use GR3 instead of initial model | GRPO / GR3 · AIME25, R1-Distill-7B | Avg@32 and tokens: 39.4 -> 46.9; 14032 -> 8582 | Improved |
| use GR3 instead of GRPO on 1.5B | GRPO / GR3 · AIME24, R1-Distill-1.5B | Avg@32: 39.6 -> 45.2 | Improved |
| use GR3 instead of GRPO on code | GRPO / GR3 · LiveCodeBench v6, R1-Distill-7B | score: 42.4 -> 41.6; tokens 10956 -> 7504 | Mixed |
| use GR3 instead of GRPO in RLHF | GRPO / GR3 · Arena-Hard-Auto, Qwen3-8B | tokens: 2343 -> 1178 | Improved |
| use GR3 instead of GRPO in RLHF | GRPO / GR3 · AlpacaEval, Qwen3-4B | score: 33.9 -> 44.1 | Improved |
| use threshold truncation (4k) instead of GRPO | GRPO / GR3 · AIME24, R1-Distill-1.5B step 800 | score: 41.0 -> 32.5 | Hurt |
| use Efficiently group-relative penalty alpha 0.2 instead of GRPO | GRPO / GR3 · MATH500, R1-Distill-1.5B | score: 87.2 -> 60.5 | Hurt |
| increase length penalty alpha from 0.33 to 1.00 (ablation) | GRPO / GR3 · AIME24, R1-Distill-1.5B | Avg@32: 45.2 -> 34.9 | Hurt |
| decrease alpha from 0.33 to 0.10 (ablation) | GRPO / GR3 · AIME24, R1-Distill-1.5B | Avg@32: 45.2 -> 43.2 | Hurt |
| decrease alpha from 0.33 to 0.20 (ablation) | GRPO / GR3 · AIME24, R1-Distill-1.5B | Avg@32: 45.2 -> 45.1 | No clear effect |
Closest papers by meaning
- Post-hoc Reward Calibration: A Case Study on Length Bias (ICLR 2025 · similarity 0.88)
- Group-in-Group Policy Optimization for LLM Agent Training (NeurIPS 2025 · similarity 0.87)
- Unveiling Implicit Advantage Symmetry: Why GRPO Struggles with Exploration and Difficulty Adaptation (NeurIPS 2026 · similarity 0.87)
- On the Effect of Negative Gradient in Group Relative Deep Reinforcement Optimization (NeurIPS 2025 · similarity 0.86)
- TGRL: Temperature-Grouped Reinforcement Learning for Efficient Exploration in LLMs (NeurIPS 2026 · similarity 0.86)
- ODIN: Disentangled Reward Mitigates Hacking in RLHF (ICML 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
