Research map / Large language models / Agents, code and math
CodeMMLU: A Multi-Task Benchmark for Assessing Code Understanding & Reasoning Capabilities of CodeLLMs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces CodeMMLU, a ~20k-question multiple-choice benchmark for code understanding and reasoning across knowledge and fundamental coding tasks, evaluating 40+ code and general LLMs.
Problem
Existing code benchmarks focus on open-ended generation and rely on test execution or LLM judges, offering limited assessment of whether CodeLLMs truly understand code.
Approach
A large-scale multiple-choice benchmark spanning software-knowledge questions (syntactic and semantic) and fundamental coding tasks (code completion, fill-in-the-blank, code repair, execution prediction), with LLM-based and execution-based filtering, distractor generation, and answer-order permutation to probe selection bias.
Key results
- GPT o3-mini achieves the top CodeMMLU average of 62.36%, with Claude 3.7 Sonnet at 61.65%.
- Chain-of-thought prompting significantly lowers GPT-4o performance on CodeMMLU, while few-shot prompting is the most reliable strategy.
- Model rankings on the knowledge test correlate with real-world fundamental-task performance at Pearson r = 0.61.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Chain-of-Thought prompting instead of zero-shot prompting | GPT-4o · CodeMMLU | accuracy | Hurt |
| Use Chain-of-Thought prompting instead of zero-shot prompting | CodeLLMs across evaluated model families · CodeMMLU | accuracy | Hurt |
| Use few-shot prompting instead of zero-shot prompting | CodeLLMs · CodeMMLU knowledge tasks | accuracy | Helped |
| Use few-shot prompting instead of zero-shot prompting | CodeLLMs · CodeMMLU real tasks | accuracy | Hurt |
| Instruction-tune DeepSeek-Coder-33B | DeepSeek-Coder-33B · CodeMMLU | accuracy: approximately 29% | Helped |
| Use DeepSeek R1 instead of DeepSeek V3 | DeepSeek · CodeMMLU | accuracy (%) — overall: 49.08 -> 43.91 | Hurt |
| Move the correct answer from position A to position B | CodeLlama-7B-Python · CodeMMLU code completion MCQ | accuracy (%): 0.00 -> 90.24 | Helped |
| Move the correct answer from position B to position A | CodeLlama-7B-Python · CodeMMLU code completion MCQ | accuracy (%): 90.24 -> 0.00 | Hurt |
| Reorder the correct answer among A, B, C, and D | LLMs on CodeMMLU code completion · CodeMMLU code completion MCQ | accuracy: average performance drop of 25%; average increase of 15.49% | Mixed |
| Use open-ended generation rather than MCQ code completion | CodeLLMs · HumanEval and CodeMMLU code completion | accuracy | Mixed |
| Increase answer-choice order sensitivity | CodeLLMs on CodeMMLU code completion · CodeMMLU code completion MCQ | accuracy standard deviation: 4.38; 31.75 | Mixed |
Surprising findings
- DeepSeek R1 underperforms its base model DeepSeek V3 on CodeMMLU (43.91 vs 49.08), and reasoning models generally overreason.
- Models strong on HumanEval often fail the MCQ version of the same problems, and accuracy shifts sharply with the position of the correct answer.
Limitations
- Data leakage cannot be fully ruled out since sources are crawled web content; mitigation is partial (perplexity and n-gram analysis).
- Answer extraction relies on regex parsing and MCQ format may not reflect open-ended coding ability.
- LLM-based filtering uses LLM judges (GPT-3.5, Llama, Mixtral) whose reliability is only spot-checked on 100 instances per subject.
Experimental setup
Datasets: W3Schools quizzes GeeksforGeeks Common Crawl HumanEval QuixBugs IBM CodeNet LeetCode
Benchmarks: CodeMMLU HumanEval MMLU GSM8K MBPP
Metrics: accuracy standard deviation across answer positions Pearson correlation perplexity 5-gram accuracy
Baselines: HumanEval MBPP CRUXEval LiveCodeBench CodeApex CodeScope
Builds on: MMLU HumanEval QuixBugs IBM CodeNet
Closest papers by meaning
- DyCodeEval: Dynamic Benchmarking of Reasoning Capabilities in Code Large Language Models Under Data Contamination (ICML 2025 · similarity 0.91)
- CoRe: Benchmarking LLMs’ Code Reasoning Capabilities through Static Analysis Tasks (NeurIPS 2025 · similarity 0.90)
- StackEval: Benchmarking LLMs in Coding Assistance (NeurIPS 2024 · similarity 0.90)
- SemCoder: Training Code Language Models with Comprehensive Semantics Reasoning (NeurIPS 2024 · similarity 0.90)
- BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions (ICLR 2025 · similarity 0.89)
- LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.