Research map / Large language models / Reasoning and chain of thought
Get an A in Math: Progressive Rectification Prompting
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Progressive Rectification Prompting (PRP), a zero-shot verify-then-rectify loop that masks a question number, checks whether the LLM can recover it from the candidate answer, and re-solves with incorrect answers as negative hints, reaching 90.5 average accuracy on eight MWP datasets with text-davinci-003.
Problem
Chain-of-thought prompting for math word problems is brittle: a single mistake in the reasoning path yields a wrong answer, and self-consistency and progressive hints repeat the same errors.
Approach
Verify each answer via substitute verification (mask a question number, supply the answer as a condition, and check whether the LLM recovers the masked value), then rectify by re-solving with a negation hint listing previously rejected answers; iterate up to K=5 times.
Key results
- PRP reaches 90.5 average accuracy over eight MWP datasets with text-davinci-003, versus 77.3 for the best zero-shot baseline (Zero-Shot-CoT) and 81.0 for the best few-shot baseline (PHP-CoT).
- On GSM-IC2-1K, PRP improves from 70.7 (initial answer) to 93.1 after five iterations.
- PRP beats Zero-Shot-CoT with self-consistency (M=10) on GSM8K (73.6 vs 70.7) and SVAMP (86.2 vs 81.7).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Progressive rectification (verify-then-rectify iterations) instead of single zero-shot CoT answer | text-davinci-003 zero-shot solving · 8 MWP datasets average | accuracy: 77.3 -> 90.5 | Helped |
| Progressive rectification instead of self-consistency (10 samples) (ablation) | Zero-Shot-CoT · GSM8K and SVAMP | accuracy: +2.9 on GSM8K, +4.5 on SVAMP | Helped |
| Substitute verification instead of enumeration verification for gold-answer identification (ablation) | verification module · AddSub and SingleEq | accuracy of gold answer identification: +13.7 on AddSub, +10.7 on SingleEq | Helped |
| Iterating verify-then-rectify for 5 iterations on GSM-IC2-1K (ablation) | PRP · GSM-IC2-1K | accuracy: 70.7 -> 93.1 | Helped |
| Iterative progressive hints (PHP-CoT) over five iterations (ablation) | PHP-CoT · GSM-IC2-1K | accuracy: 73.9 -> 75.3 | No clear effect |
| Using text-davinci-003 instead of text-davinci-002 as backbone for PRP (ablation) | PRP · SingleEq | accuracy: +12.2 accuracy | Helped |
| Zero-shot PRP vs few-shot PHP-CoT on GSM8K/GSM-IC/GSM-ICM | PRP · GSM8K, GSM-IC2-1K, GSM-ICM-1K | accuracy: +16.1, +17.8, +26.2 over PHP-CoT | Helped |
| PRP's iteration count is fixed at max K=5 (trade-off setting) (ablation) | PRP · 8 MWP datasets | accuracy | No clear effect |
Surprising findings
- Substitute verification with a single randomly chosen masked number outperformed enumerating all numbers on gold-answer identification (88.2 vs 74.5 on AddSub), while being cheaper in API calls.
- Zero-shot PRP outperforms few-shot methods, indicating manually designed demonstrations are not needed for strong MWP accuracy.
Limitations
- Evaluated only on text-davinci-003 (and davinci-002 in an ablation), so generality to other LLMs is untested.
- Verification relies on the LLM's ability to recover masked numbers; it can accept wrong answers or reject correct ones.
- Iteration count capped at K=5 for efficiency, with more iterations possibly improving results.
Experimental setup
Datasets: AddSub SingleOp MultiArith SingleEq SVAMP GSM8K GSM-IC2-1K GSM-ICM-1K
Benchmarks: AddSub SingleOp MultiArith SingleEq SVAMP GSM8K GSM-IC2-1K GSM-ICM-1K
Metrics: accuracy
Baselines: Direct Zero-Shot-CoT PS Manual-CoT Auto-CoT PHP-CoT Zero-Shot-CoT + SC
Builds on: chain-of-thought prompting zero-shot CoT progressive-hint prompting self-consistency Plan-and-Solve prompting
Closest papers by meaning
- Boosting of Thoughts: Trial-and-Error Problem Solving with Large Language Models (ICLR 2024 · similarity 0.84)
- Physics of Language Models: Part 2.2, How to Learn From Mistakes on Grade-School Math Problems (ICLR 2025 · similarity 0.84)
- Enhancing Chain of Thought Prompting in Large Language Models via Reasoning Patterns (AAAI 2025 · similarity 0.84)
- Large Language Models as Analogical Reasoners (ICLR 2024 · similarity 0.84)
- To CoT or not to CoT? Chain-of-thought helps mainly on math and symbolic reasoning (ICLR 2025 · similarity 0.84)
- Premise-Augmented Reasoning Chains Improve Error Identification in Math reasoning with LLMs (ICML 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.