Research map / Large language models / Reasoning and chain of thought
SBSC: Step-by-Step Coding for Improving Mathematical Olympiad Performance
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SBSC, a multi-turn code-generation reasoning framework where an LLM iteratively generates sub-task programs that reuse prior execution outputs, improving olympiad-level math accuracy over PAL and TIR-ToRA.
Problem
LLMs struggle with competition and Olympiad-level math because single-program reasoning (PAL, TIR-ToRA) cannot systematically handle the many intermediate steps, constraints and edge cases these problems require.
Approach
Decompose problem solving into a multi-turn sequence where each turn generates a sub-task and its Python program conditioned on prior sub-tasks, programs and execution outputs, so each step can be explored, corrected and appended incrementally rather than solved in one code block.
Key results
- With Claude-3.5-Sonnet (greedy), SBSC reaches 49.33% on AMC vs 38.59% for TIR-ToRA and 39.86% on MathOdyssey vs 27.23% for TIR-ToRA.
- With GPT-4o (greedy), SBSC reaches 30.7% on AIME vs 24.63% for PAL.
- SBSC greedy decoding with Claude-3.5-Sonnet reports 35.45% on AIME, above TIR-ToRA maj@7 (26.67%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use SBSC instead of TIR-ToRA | Claude-3.5-Sonnet · AMC | accuracy: 38.59 -> 49.33 | Helped |
| use SBSC instead of PAL | Claude-3.5-Sonnet · AIME | accuracy: 27.48 -> 35.45 | Helped |
| use SBSC instead of TIR-ToRA | Claude-3.5-Sonnet · MathOdyssey | accuracy: 27.23 -> 39.86 | Helped |
| use SBSC instead of TIR-ToRA | Claude-3.5-Sonnet · OlympiadBench | accuracy: 47.69 -> 53.31 | Helped |
| use SBSC instead of TIR-ToRA | GPT-4o · AMC | accuracy: 37.33 -> 44.55 | Helped |
| use SBSC instead of PAL | GPT-4o · AIME | accuracy: 24.63 -> 30.7 | Helped |
| use SBSC instead of TIR-ToRA | GPT-4o · MathOdyssey | accuracy: 19.59 -> 26.55 | Helped |
| use SBSC instead of TIR-ToRA | GPT-4o · OlympiadBench | accuracy: 43.32 -> 48.74 | Helped |
| add natural-language comments to code exemplars (ablation) | SBSC with Claude-3.5-Sonnet · AMC 3 Yrs | accuracy: 62 -> 67 | Helped |
| add natural-language comments to code exemplars (ablation) | SBSC with Claude-3.5-Sonnet · AIME 3 Yrs | accuracy: 19 -> 27 | Helped |
| add instruction to correct the current step after code errors, without restarting from Step 1 (ablation) | SBSC with Claude-3.5-Sonnet · AMC 3 Yrs | accuracy: 60 -> 67 | Helped |
| add instruction to correct the current step after code errors, without restarting from Step 1 (ablation) | SBSC with Claude-3.5-Sonnet · AIME 3 Yrs | accuracy: 16 -> 27 | Helped |
Surprising findings
- Single-chain greedy SBSC outperforms self-consistency (maj@7) results of existing methods on several benchmarks.
- SBSC on OlympiadBench with GPT-4o is slightly below TIR-ToRA maj@7 (48.74 vs 49.61).
Limitations
- SBSC uses about 3-7x more tokens and cost per question than TIR-ToRA or PAL.
- Exemplars and system prompt are hand-curated, with a small pool of 10 exemplars and only 4 used per prompt.
- Evaluation of open-source models is limited to DeepSeekCoder V2.5 via API and does not include a training component.
Experimental setup
Datasets: AIME (last 11 years) AMC-12 (last 11 years) MathOdyssey Olympiad subset OlympiadBench math subset JEE-Bench Omni-MATH
Benchmarks: AMC-12 AIME MathOdyssey OlympiadBench JEE-Bench Omni-MATH
Metrics: accuracy majority@7 accuracy
Baselines: COT PAL TIR-ToRA L2M-PAL
Builds on: PAL ToRA Chain-of-Thought self-consistency
Closest papers by meaning
- Leveraging Online Olympiad-Level Math Problems for LLMs Training and Contamination-Resistant Evaluation (ICML 2025 · similarity 0.87)
- MathCoder: Seamless Code Integration in LLMs for Enhanced Mathematical Reasoning (ICLR 2024 · similarity 0.87)
- MathCoder2: Better Math Reasoning from Continued Pretraining on Model-translated Mathematical Code (ICLR 2025 · similarity 0.86)
- Omni-MATH: A Universal Olympiad Level Mathematic Benchmark for Large Language Models (ICLR 2025 · similarity 0.86)
- Proving Olympiad Inequalities by Synergizing LLMs and Symbolic Reasoning (ICLR 2025 · similarity 0.86)
- UGMathBench: A Diverse and Dynamic Benchmark for Undergraduate-Level Mathematical Reasoning with Large Language Models (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.