Research map / Large language models / Agents, code and math
Reasoning Through Execution: Unifying Process and Outcome Rewards for Code Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ORPS, an inference-only tree search where an LLM refines code using execution/profiling metrics and self-critique scores as process rewards, avoiding trained PRMs.
Problem
Code generation for complex algorithmic tasks where outcome supervision ignores intermediate reasoning and learned process reward models are costly and misaligned.
Approach
Unifies process and outcome supervision by beam-searching over reasoning-plus-code states, scoring candidates with a weighted sum of LLM self-critique reward (process) and execution/profiling metrics (outcome), with no PRM training.
Key results
- ORPS with self-generated tests reaches 59.9% Pass@1 on LBPP with Qwen-2.5-Coder-7B vs 40.1% for CoT and 35.8% for LDB.
- With dataset-provided unit tests (w/ T), ORPS reaches 88.9% Pass@1 on LBPP with GPT-4o-Mini and 97.6% on HumanEval.
- Reports 26.9% higher correctness and 42.2% improved code efficiency on average across 5 models and 3 benchmarks.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply ORPS instead of BoN | Qwen-2.5-Coder-7B-Instruct · LBPP | Pass@1: 53.1 -> 59.9 | Helped |
| Apply ORPS instead of LDB (w/~T) | Qwen-2.5-Coder-7B-Instruct · HumanEval | Pass@1: 87.8 -> 79.9 | Hurt |
| Apply ORPS instead of BoN | Qwen-2.5-Coder-7B-Instruct · MBPP | Pass@1: 82.9 -> 76.7 | Hurt |
| Add dataset-provided unit tests | ORPS with Qwen-2.5-Coder-7B-Instruct · LBPP | Pass@1: 59.9 -> 77.8 | Helped |
| Add dataset-provided unit tests | ORPS with Qwen-2.5-Coder-7B-Instruct · HumanEval | Pass@1: 79.9 -> 96.3 | Helped |
| Add execution feedback (ablation) | ORPS with Qwen-2.5-Coder-7B-Instruct · LBPP | Pass@1: 43.8 -> 59.9 | Helped |
| Add explicit reasoning before generating code (ablation) | ORPS with Qwen-2.5-Coder-7B-Instruct · LBPP | Pass@1: 55.6 -> 59.9 | Helped |
| Use inference-only outcome-level reward instead of trained outcome-level reward | Qwen-2.5-Coder-7B-Instruct · LBPP | Pass@1: 37.0 -> 59.9 | Helped |
| Use inference-only line-level reward instead of trained line-level reward | Qwen-2.5-Coder-7B-Instruct · LBPP | Pass@1: 32.1 -> 38.3 | Helped |
| Apply ORPS rather than PRM-GPT | Qwen-2.5-Coder-7B-Instruct · LBPP (20 Calls) | Pass@1: 44.4 -> 48.4 | Helped |
| Apply ORPS rather than REx | Qwen-2.5-Coder-7B-Instruct · LBPP (100 Calls) | Pass@1: 54.3 -> 64.2 | Helped |
| Apply ORPS rather than REx | Qwen-2.5-Coder-7B-Instruct · CodeContests (100 Calls) | Pass@1: 13.33 -> 20.61 | Helped |
Surprising findings
- Trained PRMs (GPT-4o-labeled or human-annotated) underperform the inference-only ORPS approach under the same LLM call budget.
- Removing execution feedback drops Pass@1 by 16.1 points on LBPP with Qwen-7B, more than removing reasoning (4.3 points).
Limitations
- Relies on LLM-generated unit tests that are weaker; gains are much larger when gold dataset tests are provided, which is not a fair setting in general.
- Slight disadvantage on MBPP versus Best-of-N; benchmarks may be contaminated.
- Heavy inference cost (2×N×(K×T+1) LLM calls).
Experimental setup
Datasets: LBPP HumanEval MBPP CodeContests
Benchmarks: LBPP HumanEval MBPP CodeContests
Metrics: Pass@1 average tests passed valid rate relative execution time
Baselines: CoT Reflexion LDB Best-of-N sampling REx PRM-GPT PRM-Human
Builds on: Reflexion LDB REx Best-of-N sampling
Closest papers by meaning
- Chain of Execution Supervision Promotes General Reasoning in Large Language Models (NeurIPS 2025 · similarity 0.88)
- Rewarding Progress: Scaling Automated Process Verifiers for LLM Reasoning (ICLR 2025 · similarity 0.88)
- StepCodeReasoner: Aligning Code Reasoning with Stepwise Execution Traces via Reinforcement Learning (ICML 2026 · similarity 0.88)
- NExT: Teaching Large Language Models to Reason about Code Execution (ICML 2024 · similarity 0.87)
- Revisiting Chain-of-Thought in Code Generation: Do Language Models Need to Learn Reasoning before Coding? (ICML 2025 · similarity 0.87)
- From Mathematical Reasoning to Code: Generalization of Process Reward Models in Test-Time Scaling (AAAI 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.