Research map / Reinforcement learning / RL from feedback
The Choice of Divergence: A Neglected Key to Mitigating Diversity Collapse in Reinforcement Learning with Verifiable Reward
Long Li, Zhijian Zhou, JIARAN HAO, Jason Klein Liu, Yanting Miao, Wei Pang, Xiaoyu Tan, Wei Chu, Zhe Wang, Shirui Pan, Chao Qu, Yuan Qi
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace reverse-KL with forward-KL (DPH-F) on D_pef (ablation) | Llama-3.1-8B GRPO · Spider Pass@16 | Pass@16: 80.6 -> 85.7 | Improved |
| use reverse-KL penalty instead of no KL (ablation) | GRPO · out-of-domain math (SQL-trained) | average Pass@k: 52.37 -> 48.45 | Hurt |
| add DPH-F divergence rehearsal (ablation) | GRPO · out-of-domain math (SQL-trained Llama) | average Pass@k: 52.37 -> 60.98 | Improved |
| add DPH-JS to Llama on Bird (ablation) | GRPO · Bird Pass@16 | Pass@16: 67.7 -> 72.4 | Improved |
| increase eta of f-divergence loss from 0.01 to 0.2 (ablation) | DPH-JS (Generator) · Bird Pass@16 | Pass@16: 68.1 -> 72.4 | Improved |
| use Divergence Definition form instead of Generator form (ablation) | DPH-JS · Bird/Spider | Pass@16: 72.4 -> 71.9 (Bird) | No clear effect |
| increase alpha in alpha-divergence from 0.2 to 0.8 (ablation) | DPH alpha-divergence · Bird Pass@16 | Pass@16: 69.1 -> 70.4 | Improved |
| use D_pef threshold 8 of 8 vs 7 of 8 (ablation) | DPH-JS · Bird Pass@16 | Pass@16: 73.5 vs 73.6 | No clear effect |
| use full data for D_pef instead of thresholded split (ablation) | DPH-JS · Bird greedy | Greedy: 62.8 -> 57.4 | Hurt |
| use full data for D_pef instead of thresholded split (ablation) | DPH-JS · Bird Pass@16 | Pass@16: 72.4 -> 76.6 | Improved |
| add DPH-JS to Qwen2.5-Math-7B | GRPO · Math average Pass@k | average Pass@k: 67.71 -> 70.74 | Improved |
| add DPH-JS to OmniSQL-32B | GRPO · Bird Pass@16 | Pass@16: 78.8 -> 81.9 | Improved |
Closest papers by meaning
- Whatever Remains Must Be True: Filtering Drives Reasoning in LLMs, Shaping Diversity (ICLR 2026 · similarity 0.91)
- Exploration vs Exploitation: Rethinking RLVR through Clipping, Entropy, and Spurious Reward (ICLR 2026 · similarity 0.91)
- SAGE: Shaping Anchors for Guided Exploration in RLVR of LLMs (ICML 2026 · similarity 0.91)
- The Reasoning Boundary Paradox: How Reinforcement Learning Constrains Language Models (NeurIPS 2026 · similarity 0.91)
- Diversity-Incentivized Exploration for Versatile Reasoning (ICLR 2026 · similarity 0.90)
- Reinforcement Learning for Reasoning in Large Language Models with One Training Example (NeurIPS 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
