Research map / Large language models / Agents, code and math
SemCoder: Training Code Language Models with Comprehensive Semantics Reasoning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces SemCoder, a 6.7B Code LLM trained with monologue reasoning (forward/backward verbal execution reasoning) on executable Python data (PyX) and debugging data (PyX-R), matching or beating GPT-3.5-turbo on code generation and execution reasoning.
Problem
Code LLMs trained on static text struggle to reason about execution semantics, which hurts execution prediction, debugging, and self-refinement.
Approach
Train the model to verbalize code semantics (high-level functionality, abstract key properties, and step-by-step operational state changes via forward/backward monologues) using rejection-sampled LLM annotations verified by execution, instead of concrete scratchpad traces.
Key results
- SemCoder-S achieves 79.3% HumanEval base (GPT-3.5-turbo 76.8%), and 63.6% on CRUXEval-I (GPT-3.5-turbo 50.3%).
- SemCoder reaches 65.1% on CRUXEval-O versus GPT-3.5-turbo's 59.0%, and 61.2% on LCB-Exec for SemCoder-S.
- In the ablation, monologue reasoning outperforms Scratchpad, NExT, and concise traces on CRUXEval-I (61.8 vs 48.8 best baseline).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train DeepSeekCoder 6.7B with comprehensive program semantics using SemCoder | DeepSeekCoder 6.7B · CRUXEval-I | pass@1: 39.5 -> 62.5 | Helped |
| Train DeepSeekCoder 6.7B with comprehensive program semantics using SemCoder | DeepSeekCoder 6.7B · CRUXEval-O | pass@1: 41.2 -> 65.1 | Helped |
| Train DeepSeekCoder 6.7B with comprehensive program semantics using SemCoder | DeepSeekCoder 6.7B · LiveCodeBench-Exec | pass@1: 36.1 -> 59.7 | Helped |
| Use monologue reasoning rather than few-shot prompting (ablation) | DeepSeekCoder 6.7B · CRUXEval-I | pass@1: 39.5 -> 61.8 | Helped |
| Use monologue reasoning rather than Scratchpad (ablation) | DeepSeekCoder 6.7B · CRUXEval-O | pass@1: 50.6 -> 63.5 | Helped |
| Use monologue reasoning rather than NeXT trace format (ablation) | DeepSeekCoder 6.7B · LiveCodeBench-Exec | pass@1: 32.2 -> 58.5 | Helped |
| Use monologue reasoning rather than concise concrete traces (ablation) | DeepSeekCoder 6.7B · CRUXEval-I | pass@1: 52.1 -> 61.8 | Helped |
| Add extended Evol-instruct and partial CodeContest training data | SemCoder · LiveCodeBench-Lite | pass@1: 22.4 -> 27.5 | Helped |
| Add extended Evol-instruct and partial CodeContest training data | SemCoder · CRUXEval-O | pass@1: 65.1 -> 63.9 | Hurt |
| Fine-tune on executable PyX instead of OSS-Instruct (ablation) | DeepSeekCoder-based model · HumanEval | pass@1: 67.1 -> 70.1 | Helped |
| Fine-tune on executable PyX instead of OSS-Instruct (ablation) | DeepSeekCoder-based model · MBPP | pass@1: 77.5 -> 78.6 | No clear effect |
Surprising findings
- Free-form chain-of-thought hardly helps execution reasoning, while monologue reasoning gives up to 21.7% absolute gain in output prediction.
- Monologue reasoning for debugging is comparably effective to attaching real execution traces, without needing dynamic tracing.
Limitations
- Intermediate monologue reasoning steps are occasionally flawed even when final outputs are correct.
- Relies on a stronger LLM (GPT-3.5-turbo / GPT-4o-mini) for monologue annotation due to small base model.
- Training on execution reasoning does not directly improve code generation; self-verification via monologue is left to future work.
Experimental setup
Datasets: PyX PyX-R Evol-Instruct CodeContest OSS-Instruct
Benchmarks: HumanEval MBPP LiveCodeBench-Lite CRUXEval LiveCodeBench-Exec EvalPlus
Metrics: pass@1
Baselines: GPT-3.5-Turbo CodeLlama StarCoder2 DeepSeekCoder Magicoder Llama-3.1-Instruct-8B Scratchpad NExT
Builds on: OSS-Instruct DeepSeek-Coder Scratchpad NExT Magicoder
Closest papers by meaning
- CoRe: Benchmarking LLMs’ Code Reasoning Capabilities through Static Analysis Tasks (NeurIPS 2025 · similarity 0.91)
- NExT: Teaching Large Language Models to Reason about Code Execution (ICML 2024 · similarity 0.90)
- BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions (ICLR 2025 · similarity 0.90)
- CodeMMLU: A Multi-Task Benchmark for Assessing Code Understanding & Reasoning Capabilities of CodeLLMs (ICLR 2025 · similarity 0.90)
- Chain of Execution Supervision Promotes General Reasoning in Large Language Models (NeurIPS 2025 · similarity 0.89)
- Steering Large Language Models between Code Execution and Textual Reasoning (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.