Research map / Reinforcement learning / RL from feedback
BandPO: Bridging Trust Regions and Ratio Clipping via Probability-Aware Bounds for LLM Reinforcement Learning
Yuan Li, Bo Wang, Yufei Gao, Yao Yuqian, Xinyuan Wang, Zhangyue Yin, Xipeng Qiu
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace fixed ratio clipping with Band probability-aware bounds (KL, delta 0.05) | GRPO · AMC2023/AIME2024/AIME2025 average, Qwen2.5-3B-Instruct | mean@32: 16.82 -> 22.16 | Improved |
| replace canonical clipping with Band | GRPO · average over three math benchmarks, 1.5B to 8B models | mean@32: at least +1.5 points on every scale | Improved |
| use Band instead of Clip-Higher | GRPO · average over three math benchmarks | mean@32: +0.45, +2.11, +2.41, +0.80 across 1.5B, 3B, 7B, 8B | Improved |
| use Band instead of DCPO-DAC | GRPO · average over three math benchmarks | mean@32: +1.39, +0.80, +1.14, +0.57 across 1.5B, 3B, 7B, 8B | Improved |
| use Band instead of canonical clipping | GRPO · average pass@32, 1.5B, 3B, 8B models | pass@32: +5.13, +10.38, +3.48 | Improved |
| use Band instead of canonical clipping | GRPO · average pass@32, DeepSeek-R1-Distill-Qwen-7B | pass@32: 66.76 -> 66.05 | No clear effect |
| heuristically relax the Band bound to cover the Clip-Higher range (ablation) | BandPO (KL, 0.05) · average over three math benchmarks | mean@32: original Band better by +1.07, +1.91, +0.96, +0.61 (1.5B, 3B, 7B, 8B) | Hurt |
| sweep trust-region radius delta among 0.03, 0.05, 0.10 (ablation) | BandPO, Qwen2.5-3B-Instruct · average over three math benchmarks | mean@32 / pass@32: 0.05 gives 22.16/42.32 vs 20.28 (0.03) and 20.35 (0.10) mean@32 | Improved |
| sweep trust-region radius delta among 0.03, 0.05, 0.10 (ablation) | BandPO, DeepSeek-R1-Distill-Qwen-7B · average over three math benchmarks | mean@32: 49.78 / 50.71 / 50.28 for 0.03 / 0.05 / 0.10 | No clear effect |
| use Band bounds instead of fixed clipping for low-probability tokens | GRPO, Qwen2.5-3B · clip-high rate for tokens with p < 0.2 and positive advantage | clip rate: 20-60% -> approximately 0% | Improved |
| use Band bounds instead of fixed clipping | GRPO, Qwen2.5-3B · policy entropy during training | entropy | Improved |
| use Band instead of fixed clipping | GRPO, Qwen2.5-3B-Instruct · end-to-end training time | total hours: 39.67 -> 37.56 hours | Improved |
Closest papers by meaning
- Simple Policy Optimization (ICML 2025 · similarity 0.88)
- TIC-GRPO: Provable and Efficient Optimization for Reinforcement Learning from Human Feedback (NeurIPS 2026 · similarity 0.87)
- STAPO: Stabilizing Reinforcement Learning for LLMs by Silencing Rare Spurious Tokens (NeurIPS 2026 · similarity 0.86)
- Embedding Safety into RL: A New Take on Trust Region Methods (ICML 2025 · similarity 0.86)
- STARE: Surprisal-Guided Token-Level Advantage Reweighting for Policy Entropy Stability (NeurIPS 2026 · similarity 0.86)
- Multi-Step Likelihood-Ratio Correction for Reinforcement Learning with Verifiable Rewards (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
