Research map / Reinforcement learning / RL from feedback
GVPO: Group Variance Policy Optimization for Large Language Model Post-Training
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes GVPO, a post-training objective whose gradient weights use the KL-constrained closed-form optimal policy with a zero-sum group weighting, yielding a unique optimum and allowing arbitrary sampling distributions, evaluated on math reasoning with Qwen2.5-Math models and a summarization alignment task.
Problem
GRPO-style post-training is unstable and sensitive to hyperparameters (clip threshold, KL coefficient), and policy-gradient methods require on-policy sampling or importance sampling, while DPO may not converge to the KL-constrained optimum.
Approach
Exploit the closed-form optimal policy of KL-constrained reward maximization; because group weights sum to zero within a prompt group, the partition function Z(x) cancels, so the loss becomes a mean-squared error between centered implicit rewards and centered actual rewards. This gives a unique optimum and permits any sampling distribution with full support over the reference policy's support.
Key results
- On Qwen2.5-Math-7B, GVPO reaches 20.72 on AIME2024 and 83.80 on MATH500 versus 14.79 and 80.00 for GRPO.
- Across 10 seeds on Qwen2.5-Math-1.5B, GVPO averages 15.56 on AIME2024 vs 13.59 for GRPO, with comparable standard deviations.
- On Llama-3.1-8B-Instruct, GVPO improves AIME2024 from 8.54 (GRPO) to 11.56 and MATH500 from 52.60 to 56.60.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use GVPO instead of Remax | Qwen2.5-Math-7B · AIME2024 | avg@32 accuracy: 17.19 -> 20.72 | Helped |
| Use GVPO instead of Remax | Qwen2.5-Math-7B · AMC | pass@1 accuracy: 60.24 -> 62.65 | Helped |
| Use GVPO instead of Remax | Qwen2.5-Math-7B · MATH500 | pass@1 accuracy: 82.00 -> 83.80 | Helped |
| Use GVPO instead of Dr.GRPO | Qwen2.5-Math-7B · Minerva | pass@1 accuracy: 44.48 -> 45.95 | Helped |
| Use GVPO instead of Remax | Qwen2.5-Math-7B · Olympiadbench | pass@1 accuracy: 45.19 -> 46.96 | Helped |
| Use GVPO instead of GRPO | Llama-3.1-8B-Instruct · MATH500 | pass@1 accuracy: 52.60 -> 56.60 | Helped |
| Use GVPO instead of DPO | Qwen2.5 1.5B · Reddit summarization test | average reward: 4.83 -> 5.75 | Helped |
| Use GVPO instead of DPO | Qwen2.5 1.5B · Reddit summarization test | preference accuracy: 60.43% -> 64.93% | Helped |
| add the current-policy variance regularizer (ablation) | GVPO · AIME24 | accuracy: 0.00 -> 20.72 | Helped |
| add the policy-reference covariance regularizer (ablation) | GVPO · AIME24 | accuracy: 0.00 -> 20.72 | Helped |
| add both variance and covariance regularizers (ablation) | GVPO · AIME24 | accuracy: 7.19 -> 20.72 | Helped |
| replace entropy regularization with the current-policy variance regularizer (ablation) | GVPO · AIME24 | accuracy: 0.00 -> 20.72 | Helped |
Surprising findings
- Removing both the variance and covariance regularization terms causes divergence after about 10% of training, and removing either term alone prevents convergence.
- Replacing the variance term with entropy regularization fails to converge or underperforms, even with a lower learning rate.
Limitations
- Ablations on the sampling distribution and k use Qwen2.5-Math-1.5B only, and the main experiments use a single run; the robustness check uses 10 seeds only on the 1.5B model.
- The summarization experiment is limited to a single 1.5B model with a small human study of 100 prompts and three workers.
- Stability claims rest on empirical comparisons with fixed hyperparameters from original papers for baselines.
Experimental setup
Datasets: Competition Math dataset Reddit summarization dataset
Benchmarks: AIME2024 AMC MATH500 Minerva OlympiadBench
Metrics: pass@1 avg@32 average reward win rate preference accuracy
Baselines: GRPO Dr.GRPO Remax Reinforce++ DPO
Builds on: GRPO DPO REINFORCE++ Dr.GRPO ReMax
Closest papers by meaning
- Group Causal Policy Optimization for Post-Training Large Language Models (AAAI 2026 · similarity 0.92)
- GDPO: Group reward-Decoupled Normalization Policy Optimization for Multi-reward RL Optimization (ICML 2026 · similarity 0.90)
- Group Robust Preference Optimization in Reward-free RLHF (NeurIPS 2024 · similarity 0.90)
- Group-in-Group Policy Optimization for LLM Agent Training (NeurIPS 2025 · similarity 0.89)
- DeepVideo-R1: Video Reinforcement Fine-Tuning via Difficulty-aware Regressive GRPO (NeurIPS 2025 · similarity 0.89)
- TIC-GRPO: Provable and Efficient Optimization for Reinforcement Learning from Human Feedback (NeurIPS 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.