Research map / Large language models / Reasoning and chain of thought
StepBack: Step-Level Error-Localized Resampling for Efficient Test-Time Reasoning
Yang Ouyang, Jung-Eun Kim
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes StepBack, a step-level error-localized resampling method that improves efficiency of test-time chain-of-thought reasoning relative to full-trajectory self-consistency.
Problem
Self-consistency for chain-of-thought reasoning requires sampling many complete trajectories under a preset budget, and existing efficient variants only stop early or reallocate samples across questions.
Approach
Instead of resampling whole trajectories, localize errors to specific reasoning steps and resample from the point of error, reducing cost while keeping the reasoning quality of self-consistency.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace full-trajectory SC with step-level error-localized resampling (PRM signal) | StepBack, 3 models · 7 reasoning and QA benchmarks | accuracy and tokens: +0.36/+0.17/+0.82 points with 18.66/21.07/22.80% fewer tokens | Improved |
| use NLL signal instead of PRM for localization (ablation) | StepBack · 7 benchmarks | token saving: 27.74/27.15/33.32% saved, accuracy -0.17/+0.12/+0.55 | Improved |
| oracle first-error localization for suffix resampling (ablation) | draft-suffix resampling · GSM8K | accuracy: 91.21 -> 91.89 with 64.0% savings | Improved |
| oracle first-error localization for suffix resampling (ablation) | draft-suffix resampling · MATH500 | accuracy: 72.80 -> 72.40 (8x4), 73.00 (16x2) | No clear effect |
| localization on harder benchmarks AIME24 | StepBack · AIME24 | accuracy: -2.22 for Llama-3B and Qwen-7B | Hurt |
| combine StepBack with Adaptive Consistency | AC, Qwen2.5-3B · GSM8K | accuracy: 91.13 -> 92.34; tokens 3,509 -> 2,166 | Improved |
| combine StepBack with AC on MATH500 | AC, Qwen2.5-3B · MATH500 | accuracy: 72.80 -> 72.60; tokens 11,386 -> 8,749 | No clear effect |
| use 1.5B PRM instead of 7B PRM (ablation) | StepBack · MATH500 | accuracy vs SC: +0.36 at 8x4; -0.36 at 16x2 (7B: +0.72) | Mixed |
| more suffix-heavy budget split (1x32, 2x16) (ablation) | StepBack · 7 benchmarks | accuracy stability and tokens | Mixed |
| more draft-heavy budget split (8x4, 16x2) (ablation) | StepBack · 7 benchmarks | accuracy | Improved |
| PRM localization vs final-step resampling (ablation) | StepBack · HotpotQA | accuracy and tokens | Hurt |
| PRM localization vs final-step resampling at 4x8 (ablation) | StepBack · 7 benchmarks | accuracy: 2.06 points more accurate | Improved |
Experimental setup
Baselines: self-consistency
Builds on: self-consistency
Closest papers by meaning
- Optimal Self-Consistency for Efficient Reasoning with Large Language Models (ICML 2026 · similarity 0.90)
- Escape Sky-high Cost: Early-stopping Self-Consistency for Multi-step Reasoning (ICLR 2024 · similarity 0.89)
- A Theoretical Study on Bridging Internal Probability and Self-Consistency for LLM Reasoning (NeurIPS 2025 · similarity 0.88)
- Sample Smart, Not Hard: Correctness-First Decoding for Better Reasoning in LLMs (ICLR 2026 · similarity 0.88)
- Diagnosing Multi-step Reasoning Failures in Black-box LLMs via Stepwise Confidence Attribution (ICML 2026 · similarity 0.88)
- V1: Unifying Generation and Self-Verification for Parallel Reasoners (ICML 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
