Research map / Large language models / Reasoning and chain of thought
Diversity-Aware Policy Optimization for Large Language Model Reasoning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that solution diversity correlates with a new Potential@k metric across 12 LLMs and proposes R1-zero-Div, which adds a token-level entropy term applied only to positive samples in GRPO, gaining 3.5% average Pass@1 on four math benchmarks.
Problem
Diversity's role in RL for LLM reasoning is underexplored; standard R1-zero/GRPO training can collapse solution diversity and entropy, limiting exploration and final reasoning performance.
Approach
A token-level entropy objective (reformulated via expected log-likelihood with importance sampling) that avoids length bias of sequence-level entropy, applied only to correct (positive) samples in the GRPO group, since the gradient for low-probability tokens would otherwise promote incorrect solutions.
Key results
- R1-zero-Div averages 64.4% Pass@1 across four math benchmarks versus 61.0% for R1-zero (+3.5 points) on Qwen2.5-Math-7B.
- Div-Equ diversity on GSM8K test rises to 79.29 for R1-zero-Div versus 75.02 for R1-zero.
- On Qwen2.5-Math-1.5B, R1-zero-Div averages 57.4% versus 56.1% for R1-zero (+2.3 points on average, improving 3 of 4 benchmarks).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add token-level diversity objective to positive samples (λ=0.01) | R1-zero with Qwen2.5-Math-7B · Four mathematical benchmarks (average) | Pass@1 accuracy: 61.0 -> 64.4 | Helped |
| Add token-level diversity objective to positive samples (λ=0.01) | R1-zero with Qwen2.5-Math-7B · GSM8K test | Div-Equ: 75.02 -> 79.29 | Helped |
| Set diversity weight to λ=0.02 on positive samples instead of λ=0.01 (ablation) | R1-zero with Qwen2.5-Math-7B · Four mathematical benchmarks (average) | Pass@1 accuracy: 64.4 -> 62.8 | Hurt |
| Set diversity weight to λ=0.05 on positive samples instead of λ=0.01 (ablation) | R1-zero with Qwen2.5-Math-7B · Four mathematical benchmarks (average) | Pass@1 accuracy: 64.4 -> 61.7 | Hurt |
| Apply diversity objective to negative samples as well as positive samples (ablation) | R1-zero with Qwen2.5-Math-7B · Four mathematical benchmarks (average) | Pass@1 accuracy: 64.4 -> 63.2 | Hurt |
| Apply positive-sample diversity objective (λ=0.01) to Qwen2.5-Math-1.5B (ablation) | R1-zero with Qwen2.5-Math-1.5B · Four mathematical benchmarks | Pass@1 accuracy: 56.1 -> 57.4 | Mixed |
| Apply positive-sample diversity objective (λ=0.01) to Qwen2.5-Math-1.5B (ablation) | R1-zero with Qwen2.5-Math-1.5B · Olympiad Bench | Pass@1 accuracy: 32.1 -> 32.0 | Hurt |
Surprising findings
- Applying the diversity objective to all samples (positive and negative) yields smaller gains (63.2) than positive-only (64.4), and direct diversity promotion increases diversity in incorrect solutions.
Limitations
- Experiments limited to 1.5B and 7B models on 8x A6000 GPUs; generalization to larger models untested.
- Training only on GSM8K with short generation length; R1-zero-Div underperforms SimpleRL-Zoo reproductions in absolute terms.
- Entropy-based diversity is statistical rather than semantic, so user-intended diversity is not captured.
Experimental setup
Datasets: GSM8K MATH
Benchmarks: GSM8K MATH500 OlympiadBench College Math GPQA Diamond
Metrics: Pass@1 Avg@8 Potential@k Div-Equ Div-N-gram Div-Self-BLEU
Baselines: R1-zero SimpleRL-Zoo Eurus-2-7B-PRIME Qwen2.5-Math-7B
Builds on: GRPO DeepSeek-R1 R1-zero DAPO SimpleRL-Zoo
Closest papers by meaning
- SetPO: Set-Level Policy Optimization for Diversity-Preserving LLM Reasoning (ICML 2026 · similarity 0.94)
- Resource-Efficient Reinforcement for Reasoning Large Language Models via Dynamic One-Shot Policy Refinement (ICML 2026 · similarity 0.92)
- Revisiting Reinforcement Learning for LLM Reasoning from A Cross-Domain Perspective (NeurIPS 2025 · similarity 0.92)
- Accelerating RL for LLM Reasoning with Optimal Advantage Regression (NeurIPS 2025 · similarity 0.92)
- Beyond English-Centric Training: How Reinforcement Learning Improves Cross-Lingual Reasoning in LLMs (ICLR 2026 · similarity 0.92)
- Do Not Let Low-Probability Tokens Over-Dominate in RL for LLMs (ICLR 2026 · similarity 0.92)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.