Research map / Large language models / Reasoning and chain of thought
Premise-Augmented Reasoning Chains Improve Error Identification in Math reasoning with LLMs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Premise-Augmented Reasoning Chains (PARC), which link each math reasoning step to its premises, and the PERL dataset, showing that verifying steps under their premises improves error identification by 6-16 points absolute.
Problem
Verifying long chain-of-thought reasoning in math is hard because errors depend on distant prior steps and distractor context reduces LLM verifier accuracy; existing reference-free methods give chain-level scores rather than step-level localized errors.
Approach
Convert linear reasoning chains into DAGs by identifying each step's premises with LLMs, verify each step only under its premises (logical check) and math-check in isolation, and add an "accumulation error" category propagated via graph traversal for steps that are locally correct but rest on flawed premises.
Key results
- Llama 3.1 70B, Qwen 2.5 32B and GPT-4o reach over 90% recall in premise identification on GSM8K.
- Model-premise error identification improves over full context for most models, e.g. GPT-4o average accuracy rises from 68.52 to 79.82 on GSM8K.
- On ProcessBench MATH split, premise-augmented verification raises F1 by up to +30.7 (Qwen 2.5 32B) and +25.7 (Llama 3.1 70B).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Verify each step under its premises instead of the full reasoning context | Qwen 2.5 32B · ProcessBench GSM8K | F1: 59.8 -> 70.0 (+10.2) | Helped |
| Verify each step under its premises instead of the full reasoning context | Qwen 2.5 7B · ProcessBench GSM8K | F1: 47.2 -> 47.0 (-0.2) | Hurt |
| Verify each step under its premises instead of the full reasoning context | Qwen 2.5 72B · ProcessBench GSM8K | F1: 75.6 -> 74.1 (-1.5) | Hurt |
| Verify each step under its premises instead of the full reasoning context | Llama 3.1 8B · ProcessBench GSM8K | F1: 23.3 -> 35.6 (+12.3) | Helped |
| Verify each step under its premises instead of the full reasoning context | Qwen 2.5 32B · ProcessBench MATH | F1: 35.9 -> 66.5 (+30.7) | Helped |
| Verify each step under its premises instead of the full reasoning context | GPT-4o · GSM8K | Avg accuracy: 68.52 -> 79.82 | Helped |
| Verify each step under its premises instead of the full reasoning context | GPT-4o · MATH | Avg accuracy: 66.52 -> 76.45 | Helped |
| Verify each step under its premises instead of the full reasoning context | GPT-4o · Orca-Math | Avg accuracy: 68.55 -> 72.96 | Helped |
| Verify each step under its premises instead of the full reasoning context | GPT-4o · MetaMathQA | Avg accuracy: 69.41 -> 79.41 | Helped |
| Evaluate premise relationships pairwise rather than jointly (ablation) | GPT-4o premise mapping · GSM8K | F1: 91.47 -> 85.55 | Hurt |
| Evaluate premise relationships pairwise rather than jointly (ablation) | GPT4o-mini premise mapping · GSM8K | F1: 79.56 -> 78.74 | Hurt |
| Replace model-generated premises with ground-truth premises (ablation) | GPT-4o error identification · GSM8K | Avg accuracy: 79.82 -> 78.74 | No clear effect |
Surprising findings
- Synthetic negatives are easier to detect than naturally occurring errors, suggesting perturbation-based benchmarks overstate verifier ability.
- Dyadic (pairwise) premise mapping lowers precision substantially compared with aggregative mapping despite higher cost.
Limitations
- Premise and error annotations come from o1-preview, with only a 10% manual check and some discrepancies found.
- Premise-based verification reduces accuracy on correct steps, especially for smaller models.
- Dyadic premise mapping over-marks premises, reducing precision.
Experimental setup
Datasets: GSM8K MATH Orca-Math MetaMathQA ProcessBench
Benchmarks: PERL ProcessBench
Metrics: precision recall F1 accuracy
Baselines: Full Context (linear reasoning chain verification) ProcessBench critic baseline
Builds on: chain-of-thought ProcessBench ROSCOE ReCEval
Closest papers by meaning
- Scaling Code-Assisted Chain-of-Thoughts and Instructions for Model Reasoning (NeurIPS 2025 · similarity 0.91)
- DAG-Math: Graph-Guided Mathematical Reasoning in LLMs (ICLR 2026 · similarity 0.90)
- Understanding Chain-of-Thought in LLMs through Information Theory (ICML 2025 · similarity 0.90)
- To CoT or not to CoT? Chain-of-thought helps mainly on math and symbolic reasoning (ICLR 2025 · similarity 0.90)
- Causal Sufficiency and Necessity Improves Chain-of-Thought Reasoning (NeurIPS 2025 · similarity 0.90)
- RMath: A Logic Reasoning-Focused Datasets Toward Mathematical Multistep Reasoning Tasks (AAAI 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.