Research map / Large language models / Agents, code and math
When Do Program-of-Thought Works for Reasoning?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CIRS, an AST- and cyclomatic-complexity-based score for code rationales, finds mid-complexity program-of-thought data best for training LLaMA-7B reasoning, and uses it to filter synthesized math instructions and Code Alpaca.
Problem
When and why program-of-thought prompting improves LLM reasoning; which code data complexity levels help LLMs learn reasoning abilities.
Approach
Defines a complexity-impacted reasoning score as the product of an AST-based structural score (node count, node types, depth) and a logical score (Halstead-style difficulty times McCabe cyclomatic complexity), and uses it to stratify synthesized code data so that mid-complexity subsets are used for fine-tuning.
Key results
- The CIRS-LLaMA-7B model reaches 97.2% on MultiArith zero-shot, above gpt-3.5-turbo's 84.8%, but scores 40.4% on GSM8K versus gpt-3.5-turbo's 65.8%.
- Mid-complexity code data yields the best accuracy across four math datasets when training LLaMA models from 7B to 65B.
- On Code Alpaca filtering, Code (CIRS)-LLaMA reaches 55.0% accuracy versus 50.0% for Code-LLaMA and 24.0% for Alpaca.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train on mid-range-complexity code data rather than low- or high-complexity code data | LLaMA models trained for program-of-thought reasoning · AsDiv, GSM8K, MultiArith, and SVAMP | accuracy | Helped |
| Increase model size from 7B to 65B | LLaMA models trained on complexity-stratified code · Mathematical reasoning datasets | accuracy | Helped |
| Use high-complexity code data instead of mid-range-complexity code data | LLMs using program-of-thought reasoning · Mathematical reasoning datasets | accuracy | Hurt |
| Replace code rationales with textual rationales (ablation) | LLaMA-7B trained on mid-range-complexity data · AsDiv, GSM8K, MultiArith, and SVAMP | accuracy | Hurt |
| Filter Code Alpaca training data using CIRS rather than using the whole training set | LLaMA-7B code generation model · Code Alpaca test | accuracy: 50.0 -> 55.0 | Helped |
Surprising findings
- Training on code rationales outperforms textual rationales on all four math datasets with the same mid-range data.
- Very high complexity data reduces performance, suggesting an inherent learnability limit for current LLMs.
Limitations
- Evaluation is limited to math reasoning and code generation; commonsense and logical reasoning are left for future work.
- Complexity bins were partly selected manually after distribution inspection, and the threshold-based clustering is loosely specified.
- Custom test sets built from random 500-instance subsets of existing benchmarks; CIRS-LLaMA lags gpt-3.5-turbo substantially on GSM8K and MATH.
Experimental setup
Datasets: AQuA GSM8K MultiArith ASDiv SVAMP MATH BIG-Bench Hard Code Alpaca
Benchmarks: AsDiv GSM8K MultiArith SVAMP MATH (level-1 subset) BigBench-Hard (algorithmic and arithmetic subset)
Metrics: accuracy
Baselines: Falcon-7B-Instruct Vicuna-7B gpt-3.5-turbo Alpaca-7B Code-LLaMA-7B
Builds on: program-of-thought chain-of-thought Halstead complexity metrics McCabe cyclomatic complexity Code Alpaca
Closest papers by meaning
- Unveiling the Magic of Code Reasoning through Hypothesis Decomposition and Amendment (ICLR 2025 · similarity 0.89)
- The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexity (NeurIPS 2025 · similarity 0.88)
- Which Data Attributes Stimulate Math and Code Reasoning? An Investigation via Influence Functions (NeurIPS 2025 · similarity 0.87)
- Revisiting Chain-of-Thought in Code Generation: Do Language Models Need to Learn Reasoning before Coding? (ICML 2025 · similarity 0.87)
- Understanding Chain-of-Thought in LLMs through Information Theory (ICML 2025 · similarity 0.87)
- Scaling Code-Assisted Chain-of-Thoughts and Instructions for Model Reasoning (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.