Research map / Large language models / Agents, code and math
McEval: Massively Multilingual Code Evaluation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces McEval, a hand-written 40-language code benchmark (~16K samples) spanning generation, explanation, and completion, plus McEval-Instruct corpus and the mCoder models fine-tuned on CodeQwen1.5 and DeepSeekCoder.
Problem
Existing code benchmarks are mostly Python-only or translate Python samples into a few other languages, lacking diversity and multilingual, multitask coverage for evaluating code LLMs.
Approach
A massively multilingual (40 languages), human-written benchmark covering generation, explanation (two-pass code-to-NL-to-code), and four completion types, paired with an LLM-generated multilingual instruction corpus using cross-lingual code transfer to fine-tune mCoder.
Key results
- GPT-4o reaches 65.2 average Pass@1 on multilingual code generation across McEval languages, the highest among evaluated models.
- mCoder (CodeQwen1.5-7B base) achieves 46.7 average generation Pass@1, competitive with DeepSeekCoder-Instruct-33B (54.3) and above CodeQwen-1.5-Chat (45.5) at 7B.
- The paper reports a persistent gap between open-source models and closed-source GPT-series models across most of the 40 languages.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune CodeQwen-1.5 on McEval-Instruct | CodeQwen-1.5 · McEval code generation (40 languages) | Pass@1 (%): 33.2 -> 46.7 | Helped |
| Use mCoder instead of GPT-4o | McEval code generation (40 languages) · McEval code generation (40 languages) | Pass@1 (%): 65.2 -> 46.7 | Hurt |
| Use mCoder instead of GPT-4 Turbo | McEval completion · McEval single-line completion (40 languages) | Pass@1 (%): 82.7 -> 78.9 | Hurt |
| Use mCoder instead of GPT-4 Turbo | McEval completion · McEval multi-line completion (40 languages) | Pass@1 (%): 76.6 -> 60.7 | Hurt |
| Use mCoder instead of GPT-4 Turbo | McEval completion · McEval span completion (40 languages) | Pass@1 (%): 75.0 -> 62.6 | Hurt |
| Use mCoder instead of GPT-4 Turbo | McEval completion · McEval span completion light (40 languages) | Pass@1 (%): 76.5 -> 63.1 | Hurt |
Surprising findings
- Scores vary widely by language; several low-resource languages (e.g., Lua, Racket, Scheme) and markup languages show notably lower Pass@1 for many models.
Limitations
- Relies on GPT-4 for draft question generation and instruction corpus creation, which may introduce bias.
- Markup languages are scored with exact match rather than execution, so metrics are not directly comparable.
Experimental setup
Datasets: McEval-Instruct open-source code instruction collection
Benchmarks: McEval
Metrics: Pass@1 (greedy) exact match for markup languages
Baselines: GPT-4o GPT-4 Turbo GPT-3.5 Turbo DeepSeekCoder-Instruct CodeQwen1.5-Chat Magicoder-S-DS CodeLlama-Instruct WizardCoder
Builds on: MultiPL-E MBXP HumanEval-X Code Llama DeepSeekCoder
Closest papers by meaning
- StackEval: Benchmarking LLMs in Coding Assistance (NeurIPS 2024 · similarity 0.89)
- BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions (ICLR 2025 · similarity 0.88)
- SciEval: A Multi-Level Large Language Model Evaluation Benchmark for Scientific Research (AAAI 2024 · similarity 0.87)
- EffiBench-X: A Multi-Language Benchmark for Measuring Efficiency of LLM-Generated Code (NeurIPS 2025 · similarity 0.87)
- CodeMMLU: A Multi-Task Benchmark for Assessing Code Understanding & Reasoning Capabilities of CodeLLMs (ICLR 2025 · similarity 0.87)
- Mercury: A Code Efficiency Benchmark for Code Large Language Models (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.