Research map / Large language models / Reasoning and chain of thought
What Makes Large Language Models Reason in (Multi-Turn) Code Generation?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Systematic grid search over reasoning, instruction, and execution-feedback prompts for single- and multi-turn code generation on CodeContests and TACO, plus rejection-sampling finetuning to internalize multi-turn CoT.
Problem
Chain-of-thought and self-repair prompting are under-explored for code generation, especially in multi-turn settings under fixed computational budgets, where it is unclear whether multi-turn approaches beat repeated single-turn sampling.
Approach
Decomposes CoT prompting for code into reasoning (NL to NL), instruction (NL to code), and execution-feedback prompts, evaluates them in a unified grid search with the compute-aware pass n@k metric, and finetunes a model via rejection sampling on correct multi-turn CoT trajectories so reasoning is internalized without inference-time prompts.
Key results
- Best CoT combination (reasoning plus instruction) raises Llama 3.1 70B pass@100 on CodeContests test from 48.9 to 54.1 in the single-turn grid search.
- Multi-turn CoT with CoT-retry lifts Llama 3.1 70B 100@300 on CodeContests test from 56.2 to 59.6, and the RFT model reaches 61.1.
- The RFT model of Llama 3.1 70B improves 1@3 on CodeContests test from 24.1 to 29.7 without CoT prompts at inference, and gains generalize to TACO test.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add tailored multi-turn CoT | Llama 3.1 70B · CodeContests test | pass 1@3: 27.3 -> 31.5 | Helped |
| Add tailored multi-turn CoT | Llama 3.1 405B · CodeContests test | pass 1@3: 32.9 -> 40.1 | Helped |
| Add multi-turn without CoT | Llama 3.1 405B · CodeContests test | pass 1@3: 32.9 -> 35.4 | Helped |
| Add single-turn CoT | Llama 3.1 405B · CodeContests test | pass 1@3: 32.9 -> 31.8 | Hurt |
| Combine the best reasoning and instruction prompts | Llama 3.1 70B · CodeContests test | pass@100: 48.9 -> 54.1 | Helped |
| Use multi-turn CoT rather than the base model | Llama 3.1 70B · TACO hard | pass n@k (%) — 1@3: 4.4 -> 5.8 | Helped |
| Use multi-turn CoT rather than the base model | Llama 3.1 70B · TACO easy | pass n@k (%) — 100@300: 60.2 -> 59.8 | Mixed |
| Add a second reasoning prompt (ablation) | Llama 3.1 70B with one reasoning and one instruction prompt · CodeContests test | pass 100@300: 59.2 -> -3.1 | Hurt |
| Add a second instruction prompt after adding a second reasoning prompt (ablation) | Llama 3.1 70B · CodeContests test | pass 100@300: -3.1 -> -2.1 | Hurt |
| Replace the simple retry prompt with explicit execution-feedback analysis and retry (ablation) | Llama 3.1 70B · CodeContests test | pass 100@300: 56.2 -> 54.6 | Hurt |
| Replace the simple retry prompt with a direct fix instruction (ablation) | Llama 3.1 8B · CodeContests test | pass 100@300: 30.4 -> 29.3 | Hurt |
| Replace the simple retry prompt with execution-feedback analysis and a direct fix instruction (ablation) | Llama 3.1 8B · CodeContests test | pass 100@300: 30.4 -> 27.5 | Hurt |
Surprising findings
- Multi-turn repeated sampling alone often underperforms single-turn repeated sampling under equal sampling budgets, and detailed execution feedback can reduce diversity.
- Many CoT prompts degrade pass@1 performance at small sampling budgets even when they help pass@100.
Limitations
- Only zero-shot evaluation on competitive programming benchmarks with Llama models and GPT-4o; GPT-4o and Llama 405B evaluated only at small sampling budgets due to compute.
- RFT assumes correct final code implies correct reasoning, with no process reward filtering.
- Does not explore tree-search or branching multi-turn strategies.
Experimental setup
Datasets: CodeContests TACO HumanEval+ MBPP+ LiveCodeBench-v4
Benchmarks: CodeContests TACO HumanEval+ MBPP+ LiveCodeBench-v4
Metrics: pass n@k pass@k pass@1 pass@100
Baselines: Single-turn repeated sampling Chain-of-thought (CoT) Multi-turn without CoT LDB feedback SFT on CodeContests
Builds on: chain-of-thought prompting LDB AlphaCode pass@k estimator Expert Iteration ReST
Closest papers by meaning
- Revisiting Chain-of-Thought in Code Generation: Do Language Models Need to Learn Reasoning before Coding? (ICML 2025 · similarity 0.90)
- Steering Large Language Models between Code Execution and Textual Reasoning (ICLR 2025 · similarity 0.89)
- Scaling Prompt Synthesis for Large Language Model Reasoning (ICML 2026 · similarity 0.89)
- ConvCodeWorld: Benchmarking Conversational Code Generation in Reproducible Feedback Environments (ICLR 2025 · similarity 0.89)
- Intention Chain-of-Thought Prompting with Dynamic Routing for Code Generation (AAAI 2026 · similarity 0.89)
- Prompt Sketching for Large Language Models (ICML 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.