Research map / Large language models / Reasoning and chain of thought
G$^2$RPO: Geometric GRPO; Escaping LLM's Reasoning Rut to Break Accuracy--Entropy Trade-off
Ali Rad, Khashayar Filom, Darioush Keivan, Peyman Mohajerin Esfahani, Ehsan Kamalinejad
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Diagnoses diversity collapse in GRPO as a dynamical flow on the probability simplex and proposes G²RPO (Geometric GRPO) to escape the resulting reasoning rut and improve the accuracy-entropy trade-off.
Problem
GRPO-style RLVR for large reasoning models often suffers diversity collapse (entropy loss), limiting exploration and creating an accuracy-entropy trade-off.
Approach
Formalizes GRPO as a dynamical flow on the probability simplex under mode-based coarse-graining of rollouts, showing it induces a geometric rut that suppresses diversity; proposes a geometry-aware modification (G²RPO) to escape it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add inverse-probability granularity bonus (G2RPO) to GRPO advantages | GRPO · AIME 2024 / 2025 | pass@1: +1.4 to +7.9 points vs GRPO | Improved |
| add G2RPO bonus to GRPO | Qwen3-14B-Base · AIME 2024 | pass@1: 45.9 -> 53.8 | Improved |
| add G2RPO bonus to GRPO | DeepSeek-R1-Distill-Qwen-7B · AIME 2025 | pass@1: 46.7 -> 48.9 | Improved |
| add G2RPO bonus to GRPO | DeepSeek-R1-Distill-Qwen-7B · mode diversity | Avg K (correct clusters): 1.97 -> 6.40 | Improved |
| train with plain GRPO instead of staying at base model | Qwen3-14B-Base · mode diversity | Avg K (correct clusters): 3.36 -> 1.45 | Hurt |
| increase gain lambda from 1.5 to 3 (ablation) | G2RPO · mode coverage | good modes discovered: +92% over lambda=1.5 | Improved |
| increase gain lambda without neutrality correction (ablation) | G2RPO · bad-mass learning | incorrect fraction p-hat decay | Hurt |
| add neutrality correction (bad-mass offset) (ablation) | G2RPO · bad-mass learning | p-hat trajectory | Improved |
| swap sentence encoder and DBSCAN scale (ablation) | G2RPO · DeepSeek-7B pass@30 | pass@30: 78% unchanged | No clear effect |
| add G2RPO bonus to GRPO | GRPO · AIME 2025 pass@k, k in {2,4,16,30} | pass@k: +1.7 to +4.3 (Qwen3-14B), +2.0 to +4.9 (DeepSeek-7B) | Improved |
| add moderate KL regularization (ablation) | G2RPO · diversity | mode diversity advantage | Mixed |
Experimental setup
Metrics: accuracy entropy
Baselines: GRPO
Builds on: GRPO
Closest papers by meaning
- TIC-GRPO: Provable and Efficient Optimization for Reinforcement Learning from Human Feedback (NeurIPS 2026 · similarity 0.88)
- S-GRPO: Early Exit via Reinforcement Learning in Reasoning Models (NeurIPS 2025 · similarity 0.87)
- Learn from the Gap: Differential-Aware Advantage Pruning with Adaptive Rollout Sampling for GRPO (NeurIPS 2026 · similarity 0.87)
- $f$-GRPO & Beyond: Divergence-Based Reinforcement Learning Algorithms for General LLM Alignment (NeurIPS 2026 · similarity 0.87)
- Your Language Model is Its Own Critic: Reinforcement Learning with Value Estimation from Actor’s Internal States (NeurIPS 2026 · similarity 0.87)
- Act Only When It Pays: Efficient Reinforcement Learning for LLM Reasoning via Selective Rollouts (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
