Research map / Large language models / Reasoning and chain of thought
GLoRe: When, Where, and How to Improve LLM Reasoning via Global and Local Refinements
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes GLoRe, a pipeline that uses an ORM to decide when to refine, a synthetic-data-trained stepwise ORM (SORM) to localize where, and global and local refinement models to decide how, improving Llama-2 13B on GSM8K.
Problem
LLMs struggle to identify when and where to refine their reasoning without external feedback; ORMs under-estimate intermediate-step correctness and PRMs require costly human step labels.
Approach
Train stepwise ORMs (SORMs) only on synthetic data by rejection-sampling the student from each prefix to approximate the optimal value function V*, then use the SORM to localize the first error for local refinement, while global refinement starts from scratch; an ORM reranks draft, global and local refinements.
Key results
- SORM improves step-level accuracy over ORM on GSM8K 13B from 0.73 to 0.81.
- Combining global and local refinements with the SORM fixes 41% of incorrect GSM8K 13B drafts (0.412), versus 0.281 for global refinement alone.
- Reranking draft, global and local refinements with the ORM improves Llama-2 13B on GSM8K from 53% to 65% greedy accuracy, per the abstract/introduction.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| train a SORM instead of an ORM for step verification | Llama-2 reward models · GSM8K test set labels | step-level accuracy: 0.73 -> 0.81 | Helped |
| train a SORM instead of an ORM for step verification | Llama-2 reward models · SVAMP test set labels | step-level accuracy: 0.85 -> 0.87 | No clear effect |
| use the SORM rather than ORM to predict final-answer correctness | Llama-2 13B reward models · GSM8K test set labels | final answer accuracy: 0.85 -> 0.8 | Hurt |
| balance positive and negative ORM labels per question (ablation) | Balanced ORM · GSM8K test set labels | step-level accuracy: 0.74 -> 0.73 | Hurt |
| use SORM instead of ORM to locate the first error for local refinement | Llama-2 13B local refinement · GSM8K incorrect drafts | refinement accuracy: 0.262 -> 0.283 | Helped |
| combine global refinement with SORM-local refinement | Llama-2 13B refinement pipeline · GSM8K incorrect drafts | refinement accuracy: 0.283 -> 0.412 | Helped |
| use global and local refinements together instead of global refinement alone | Llama-2 13B refinement pipeline · GSM8K incorrect drafts | refinement accuracy: 0.281 -> 0.412 | Helped |
| train global refinement with Expert Iteration rather than PPO (ablation) | Llama-2 7B global refiner · GSM8K test | accuracy: 0.36 -> 0.39 | Helped |
| train global refinement with PPO rather than SFT (ablation) | Llama-2 7B global refiner · GSM8K test | accuracy: 0.36 -> 0.36 | No clear effect |
| use a contrastive loss instead of classifier training for the ORM (ablation) | Llama-2 7B ORM · GSM8K test | rerank accuracy: 0.56 -> 0.47 | Hurt |
| filter SORM training data using SORM self-supervision (ablation) | SORM · SORM test set | accuracy: 75% -> 79% | Helped |
| filter SORM training data using the 2-epoch SORM checkpoint (ablation) | SORM · SORM test set | accuracy: 78% -> 79% | Helped |
Surprising findings
- Global refinement fixes a similar fraction of incorrect drafts as local refinement, despite local refinement receiving error location feedback.
- The ORM outperforms the SORM at final answer correctness, even though the SORM is better at step-level accuracy.
- A contrastive reward model reranks worse than a classifier ORM (0.47 vs 0.56).
Limitations
- Global and local refiners struggle to decide when to refine without the ORM, and bare refinement often hurts correct drafts.
- SORM labels are noisy (about 94% agreement with hand labels on 257 steps), and SORM is weaker than ORM for final answer reranking.
- Evaluation limited to two math word-problem benchmarks and Llama-2 7B/13B.
Experimental setup
Datasets: GSM8K SVAMP MATH PRM data
Benchmarks: GSM8K SVAMP
Metrics: step-level accuracy final answer accuracy refinement accuracy maj@1
Baselines: ORM Balanced ORM Best-of-3 (Bo3) PPO global refinement Prompted Llama-2 7B Value-based refinement
Builds on: Expert Iteration Outcome-based reward models (Cobbe et al.) Process-based reward models Chain-of-thought
Closest papers by meaning
- Bag of Tricks for Inference-time Computation of LLM Reasoning (NeurIPS 2025 · similarity 0.90)
- Rectifying LLM Thought from Lens of Optimization (ICLR 2026 · similarity 0.90)
- General-Reasoner: Advancing LLM Reasoning Across All Domains (NeurIPS 2025 · similarity 0.90)
- STRIDE: Learnable Stepwise Language Feedback for LLM Reasoning (NeurIPS 2026 · similarity 0.90)
- Advancing LLM Reasoning with Natural Language and Numerical Feedback (ICML 2026 · similarity 0.90)
- ReGenesis: LLMs can Grow into Reasoning Generalists via Self-Improvement (ICLR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.