Research map / Large language models / Reasoning and chain of thought
GPO: Learning from Critical Steps to Improve LLM Reasoning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes GPO, which locates the critical (max-advantage) step in a reasoning trajectory via Monte Carlo advantage estimates, resets the policy there to resample rollouts, and uses these for PPO or preference-based fine-tuning (DPO, KTO, SimPO, ORPO).
Problem
Multi-step LLM reasoning fine-tuning typically optimizes entire trajectories with trajectory-level rewards or preferences, without focusing on the intermediate steps that determine success or failure.
Approach
Identify the critical reasoning step by MC estimation of the step-level advantage, reset generation to that step, and resample continuations so that training concentrates on pivotal moments; the method is plug-in to PPO and offline preference optimizers, with a theoretical regret bound and an advantage-weighted RL interpretation.
Key results
- GPO-PPO raises MATH accuracy from 79.60% (PPO) to 87.80% and GPO-DPO from 82.40% to 86.80% on DeepSeek-R1-Distill-Qwen-7B.
- GPO improved all 5 algorithms on all 7 benchmarks in the reported table (e.g., GPO-DPO MMLUPro 48.28 to 51.93).
- On MATH, GPO-PPO achieves 87.9% versus 79.9% for Satori-style random reset; user study shows 44% to 88% agreement with GPO-identified steps per question.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add GPO critical-step selection to PPO | PPO · MATH | accuracy: 79.60 -> 87.80 | Helped |
| Add GPO critical-step selection to DPO | DPO · MATH | accuracy: 82.40 -> 86.80 | Helped |
| Add GPO critical-step selection to KTO | KTO · MATH | accuracy: 77.20 -> 79.60 | Helped |
| Add GPO critical-step selection to SimPO | SimPO · MATH | accuracy: 72.20 -> 74.00 | Helped |
| Add GPO critical-step selection to ORPO | ORPO · MATH | accuracy: 75.20 -> 78.20 | Helped |
| Select advantage-identified critical steps instead of random steps (ablation) | GPO-PPO · MATH | accuracy: 79.9% -> 87.9% | Helped |
| Group multiple generation steps into a logical step (ablation) | GPO-DPO · BBEH long-context reasoning subset | accuracy: +7% | Helped |
| Increase Monte Carlo simulations for critical-step estimation (ablation) | GPO with DPO or KTO · MATH | accuracy | Helped |
| Increase Monte Carlo simulations beyond 12 (ablation) | GPO with DPO or KTO · MATH | accuracy | No clear effect |
| Use GPO rather than the base optimization algorithm | DPO and KTO across tested model sizes · MATH | accuracy | Helped |
Surprising findings
- Random reset (Satori-style) gives substantially lower MATH accuracy than critical-step reset, suggesting the gain is not just from resetting.
Limitations
- Computational overhead from Monte Carlo advantage estimation (about 1.9x PPO training time, 1.8x data preparation).
- Critical-step quality is evaluated only indirectly via downstream accuracy and a 50-person user study; no automated metric.
- Long trajectories require heuristic step grouping; results on BBEH subset are limited to one setting.
Experimental setup
Datasets: GSM8K MATH MMLU MMLUPro BBH AIME-2024 training split (1983-2023)
Benchmarks: BBH MATH GSM8K MMLU MMLUPro AIME-2024 AIME-2025
Metrics: zero-shot pass@1 accuracy (greedy decoding)
Baselines: PPO DPO KTO SimPO ORPO Satori random reset
Builds on: PPO DPO Satori KTO SimPO
Closest papers by meaning
- General-Reasoner: Advancing LLM Reasoning Across All Domains (NeurIPS 2025 · similarity 0.91)
- ReGenesis: LLMs can Grow into Reasoning Generalists via Self-Improvement (ICLR 2025 · similarity 0.90)
- Rectifying LLM Thought from Lens of Optimization (ICLR 2026 · similarity 0.90)
- DOTS: Learning to Reason Dynamically in LLMs via Optimal Reasoning Trajectories Search (ICLR 2025 · similarity 0.90)
- A State-Transition Framework for Efficient LLM Reasoning (ICLR 2026 · similarity 0.90)
- Don’t Think Longer, Think Wisely: Optimizing Thinking Dynamics for Large Reasoning Models (NeurIPS 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.