Research map / Large language models / Reasoning and chain of thought
CRUXEval: A Benchmark for Code Reasoning, Understanding and Execution
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces CRUXEval, an 800-function Python benchmark for code input and output prediction built by generate-and-filter from Code Llama 34B, evaluates about 20 code LMs and GPT models, and tests CoT and fine-tuning.
Problem
Existing code benchmarks (HumanEval, MBPP) measure code generation, leaving code execution and understanding ability of code LMs unmeasured.
Approach
Generate a large pool of simple Python functions with input-output pairs by prompting Code Llama 34B, filter them to human-solvable short programs, and sample 800 to form a benchmark with two tasks: output prediction (CRUXEval-O) and input prediction (CRUXEval-I), with a recipe that can be rerun for harder variants.
Key results
- GPT-4 with CoT reaches 74.8% pass@1 on input prediction and 81.9% on output prediction.
- Code Llama 34B reaches 46.5% pass@1 on input and 41.1% on output without CoT; the best open model is around 47-48% on input.
- Fine-tuning Code Llama 34B on about 140K distilled samples lets it match GPT-4 on both tasks, but accuracy plateaus under 70%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add chain-of-thought prompting (ablation) | GPT-4 · CRUXEval | pass@1: +7.7% input, +18.5% output | Helped |
| add chain-of-thought prompting (ablation) | Code Llama 34B · CRUXEval | pass@1: +3.9% input, +4.9% output | Helped |
| add chain-of-thought prompting (ablation) | GPT-3.5 · CRUXEval | pass@1: -0.1% input, +13.3% output | Mixed |
| add chain-of-thought prompting (ablation) | Code Llama 13B · CRUXEval | pass@1: +0.1% input, +0.9% output | No clear effect |
| anonymize variable names to x1, x2 identifiers (ablation) | Code Llama 34B · CRUXEval subset | pass@1: +1.5% input, -2.0% output | Mixed |
| distilling on GPT-4-style data (WizardCoder, Phind, Phi) vs base Code Llama 34B | HumanEval-to-CRUXEval relationship · CRUXEval | pass@1 | No clear effect |
| fine-tune Code Llama 34B on ~140K distilled execution samples | Code Llama 34B · CRUXEval | pass@1 | Helped |
| use Code Llama 34B-generated functions vs GPT-3.5-generated functions for benchmark (ablation) | benchmark data generation · CRUXEval data-generating model ablation | pass@1 | Mixed |
Surprising findings
- Distilled models (WizardCoder, Phind, Phi) beat their base models on HumanEval but do not outperform Code Llama 34B on CRUXEval.
- CoT hurts GPT-3.5 and Code Llama 13B on input prediction, while GPT-4 benefits consistently.
Limitations
- Benchmark functions come from a single generator model (Code Llama 34B) and are short, simple, self-contained Python.
- Pass@1 and pass@k compress information; results sensitive to prompt and temperature.
- Fine-tuning decontamination only uses exact string match, so near-duplicates may remain.
Experimental setup
Datasets: Python standard library method prompts (str, list, dict) Code Llama 34B generated functions and input-output pairs
Benchmarks: CRUXEval-I CRUXEval-O HumanEval
Metrics: pass@1 pass@5 bootstrap confidence intervals
Baselines: HumanEval MBPP GPT-3.5 WizardCoder Phind v2 Phi-1 Phi-1.5 DeepSeek Coder
Builds on: HumanEval Chain-of-thought prompting Scratchpad
Closest papers by meaning
- BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions (ICLR 2025 · similarity 0.86)
- CodeIO: Condensing Reasoning Patterns via Code Input-Output Prediction (ICML 2025 · similarity 0.86)
- SemCoder: Training Code Language Models with Comprehensive Semantics Reasoning (NeurIPS 2024 · similarity 0.86)
- Chain of Execution Supervision Promotes General Reasoning in Large Language Models (NeurIPS 2025 · similarity 0.85)
- Mercury: A Code Efficiency Benchmark for Code Large Language Models (NeurIPS 2024 · similarity 0.85)
- Scaling Code-Assisted Chain-of-Thoughts and Instructions for Model Reasoning (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.