Research map / Large language models / Agents, code and math
Solving Challenging Math Word Problems Using GPT-4 Code Interpreter with Code-based Self-Verification
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes explicit code-based self-verification (CSV), a zero-shot prompt that makes GPT-4 Code Interpreter verify its answer with code and revise when verification fails, combined with verification-guided weighted majority voting, reaching 84.32% on MATH.
Problem
Math word problem solving with LLMs, where GPT-4 Code Interpreter achieves strong accuracy but lacks reliable answer verification; the paper asks whether its code generation and self-debugging can be exploited to self-verify and correct solutions without external models or users.
Approach
Analyzes how constraining code usage affects GPT4-Code accuracy (Code Usage Frequency), then uses a zero-shot code-based verification prompt so the model checks its own answer with code and revises if False, and weights majority voting by verification state (True > Uncertain > False).
Key results
- On MATH, GPT4-Code rises from 69.69% to 73.54% with CSV, and to 84.32% with CSV plus verification-guided weighted voting (k=16).
- On GSM8K, GPT4-Code + CSV + Voting reaches 97.0% using 5 sampled paths, versus 92.9% for GPT4-Code alone.
- On MMLU-Math, GPT4-Code + CSV + Voting attains 89.2% zero-shot versus 87.5% for GPT4-Code alone.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add code-based self-verification to GPT4-Code | GPT4-Code · MATH | accuracy: 69.69 -> 73.54 | Helped |
| Add verification-guided weighted majority voting to code-based self-verification | GPT4-Code + CSV · MATH | accuracy: 73.54 -> 84.32 | Helped |
| Add code-based self-verification and verification-guided voting | GPT4-Code · GSM8K | accuracy: 92.9 -> 97.0 | Helped |
| Add code-based self-verification and verification-guided voting | GPT4-Code · MMLU-Math | accuracy: 87.5 -> 89.2 | Helped |
| Add code-based self-verification and verification-guided voting | GPT4-Code · MMLU-STEM | accuracy: 86.8 -> 87.0 | Helped |
| Replace code-based self-verification with natural-language self-verification (ablation) | GPT4-Code · MATH | accuracy: 69.69 -> 69.29 | Hurt |
| Use code-based self-verification instead of no verification (ablation) | GPT4-Code · MATH | accuracy: 69.69 -> 73.54 | Helped |
| Set voting weights with true > uncertain ≥ false (ablation) | GPT4-Code + CSV + weighted voting · MATH | accuracy | Helped |
| Set weights to true=0.5, uncertain=0.5, false=1 (ablation) | GPT4-Code + CSV + weighted voting · MATH | accuracy | Hurt |
| Restrict code use to one code block instead of prohibiting code use (ablation) | GPT4-Code · MATH | accuracy: +6.9 percent | Helped |
| Prohibit code use instead of allowing code once (ablation) | GPT4-Code · MATH | accuracy: +6.9 percent | Hurt |
Surprising findings
- Natural-language self-verification slightly lowered MATH accuracy (69.29% vs 69.69%), while code-based verification improved it.
- Average precision of True-state verification is high (95.88%), suggesting accuracy could rise further if more solutions reach the True state.
Limitations
- Analysis and improvements are restricted to GPT4-Code, a closed system; not applied to other LLMs.
- Geometry gains are small (+0.6%), attributed to the need for multimodal reasoning beyond the scope of the paper.
- The MATH-code and MMLU-Math-code datasets for fine-tuning open models are described as planned rather than released.
Experimental setup
Datasets: MATH GSM8K MMLU-Math MMLU-STEM
Benchmarks: MATH GSM8K MMLU-Math MMLU-STEM
Metrics: accuracy precision recall
Baselines: GPT-4 (CoT) GPT-4 (PHP) GPT-3.5 GPT-4 Model selection Chinchilla Galactica LLaMA 2 OpenLLM
Builds on: chain-of-thought self-consistency program-aided language models self-refine
Closest papers by meaning
- MathCoder: Seamless Code Integration in LLMs for Enhanced Mathematical Reasoning (ICLR 2024 · similarity 0.91)
- JiuZhang3.0: Efficiently Improving Mathematical Reasoning by Training Small Data Synthesis Models (NeurIPS 2024 · similarity 0.88)
- MathCoder2: Better Math Reasoning from Continued Pretraining on Model-translated Mathematical Code (ICLR 2025 · similarity 0.87)
- OpenMathInstruct-2: Accelerating AI for Math with Massive Open-Source Instruction Data (ICLR 2025 · similarity 0.86)
- Scaling Code-Assisted Chain-of-Thoughts and Instructions for Model Reasoning (NeurIPS 2025 · similarity 0.86)
- MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuning (ICLR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.