Research map / Large language models / Agents, code and math
Multi-LCB: Extending LiveCodeBench to Multiple Programming Languages
Maria Ivanova, Pavel Zadorozhny, Rodion Levichev, Ivan Petrov, Adamenko Pavel, Ivan Lopatin, Alexey Kutalev, Dmitrii Babaev
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Evaluate in other languages instead of Python | OpenReasoning-Nemotron-32B · Multi-LCB v6 | Pass@1: 64.4 -> 22.7 (avg of 12 languages) | Hurt |
| Evaluate in Rust instead of Python | OpenCodeReasoning-Nemotron-1.1-32B · Multi-LCB v6 | Pass@1: 56.0 -> 1.1 | Hurt |
| Switch from Python to average over 12 languages | GPT-OSS-120B (Medium) · Multi-LCB v6 | Pass@1: 71.1 -> 67.8 | Hurt |
| Switch from Python to Rust | Qwen3-235B-A22B-Thinking-2507 · Multi-LCB v6 | Pass@1: 74.0 -> 47.7 | Hurt |
| Use GPT-OSS-120B instead of Qwen3-235B-Thinking-2507 on Rust | GPT-OSS-120B · Multi-LCB v6, Rust | Pass@1: 47.7 -> 70.5 | Improved |
| Increase reasoning effort from Low to Medium | GPT-OSS-120B · Multi-LCB v6, average | Pass@1: 53.1 -> 67.8 | Improved |
| Use thinking variant instead of Instruct variant | Qwen3-235B-A22B-2507 · Multi-LCB v6, average | Pass@1: 37.5 -> 64.0 | Improved |
| Evaluate on pre-cutoff instead of post-cutoff problems | Top-10 models · Multi-LCB monthly Pass@1 | Pass@1 | Improved |
| Convert LCB Python tasks to STDIN/STDOUT multi-language pipeline (reproduction vs original) | Qwen3-235B-A22B-Thinking-2507 · LCB v6 Python | Pass@1: 74.1 -> 74.0 | No clear effect |
| Change language from Python to Scala | All models (mean) · Multi-LCB v6 | Pass@1: 0.482 -> below 0.29 | Hurt |
Closest papers by meaning
- LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code (ICLR 2025 · similarity 0.93)
- BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions (ICLR 2025 · similarity 0.91)
- McEval: Massively Multilingual Code Evaluation (ICLR 2025 · similarity 0.90)
- EffiBench-X: A Multi-Language Benchmark for Measuring Efficiency of LLM-Generated Code (NeurIPS 2025 · similarity 0.90)
- CodeSync: Synchronizing Large Language Models with Dynamic Code Evolution at Scale (ICML 2025 · similarity 0.90)
- EvoCodeBench: An Evolving Code Generation Benchmark with Domain-Specific Evaluations (NeurIPS 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
