Research map / Large language models / Agents, code and math
LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces LiveCodeBench, a contamination-free, continuously updated benchmark of competitive programming problems from LeetCode, AtCoder, and CodeForces, evaluating four code scenarios (generation, self-repair, execution, test output prediction) across 52 LLMs.
Problem
Existing code benchmarks (HumanEval, MBPP, APPS) suffer from training-data contamination, overfitting, saturation, and focus only on natural-language-to-code generation.
Approach
Collect problems over time from competition platforms with release dates, so models can be evaluated only on problems released after their training cutoff; add self-repair, code execution, and test output prediction scenarios for holistic evaluation.
Key results
- Performance of DeepSeek-Coder and GPT-4o drops sharply on LeetCode problems released after their cutoffs, indicating contamination in older problems.
- Model scores correlate above 0.88 across scenarios, but relative rankings shift; Claude-3-Opus beats GPT-4-Turbo on test output prediction (58.7 vs 55.7 pass@1 range in table).
- DeepSeek-Coder-1.3B-Instruct reaches 59.8% pass@1 on HumanEval+ but only 26.3% on LCB-Easy, indicating HumanEval overfitting.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add post-training to the base models | Llama3-70B, DSCoder-33B, and Phind-34B · LiveCodeBench, code generation | Pass@1: 8.2, 7.3 and 9.5 points | Helped |
| Evaluate code execution with chain-of-thought prompting (ablation) | GPT-4O-2024-05-13 · LiveCodeBench, code execution | Pass@1: 39.10 -> 91.00 | Helped |
| Evaluate code execution with chain-of-thought prompting (ablation) | Claude-3-Opus · LiveCodeBench, code execution | Pass@1: 36.50 -> 80.10 | Helped |
| Evaluate code execution with chain-of-thought prompting (ablation) | Gemini-Flash-1.5-May · LiveCodeBench, code execution | Pass@1: 21.40 -> 57.10 | Helped |
| Evaluate code execution with chain-of-thought prompting (ablation) | CodeLlama-13B-Instruct · LiveCodeBench, code execution | Pass@1: 23.50 -> 14.10 | Hurt |
| Evaluate code execution with chain-of-thought prompting (ablation) | DSCoder-1.3B-Instruct · LiveCodeBench, code execution | Pass@1: 18.10 -> 17.00 | No clear effect |
| Evaluate DeepSeek models on LeetCode problems released after August 2023 | DSCoder-33B · LeetCode, code generation | Pass@1: sim 60 in May problems to Pass@1 sim 0 in September LeetCode problems | Hurt |
| Evaluate GPT-4O on problems released since its November cutoff | GPT-4O · LeetCode, code generation | performance | Hurt |
| Compare fine-tuned open models on LiveCodeBench rather than HumanEval+ | DSCoder-1.3B-Instruct · HumanEval+ and LiveCodeBench-Easy, code generation | Pass@1: 59.8% on HumanEval+ but only 26.3% on LCB-Easy | Mixed |
| Use post-training | Llama3-70B, DSCoder-33B, and Phind-34B · HumanEval+ | Pass@1 | Helped |
| Evaluate with chain-of-thought prompting (ablation) | Open models · LiveCodeBench, code execution | Pass@1 | Mixed |
| Evaluate with chain-of-thought prompting (ablation) | Closed models · LiveCodeBench, code execution | Pass@1 | Helped |
Surprising findings
- Smaller fine-tuned open models rank well on HumanEval+ yet perform much worse on LiveCodeBench, suggesting overfitting to HumanEval.
- DeepSeek-Base-33B also shows contamination, dropping from about 60 pass@1 on May problems to about 0 on September LeetCode problems.
Experimental setup
Datasets: LeetCode problems AtCoder problems CodeForces problems
Benchmarks: LiveCodeBench HumanEval+
Metrics: pass@1
Baselines: HumanEval+ GPT-4-Turbo GPT-4 Claude-3-Opus DeepSeek-Coder-33B-Instruct CodeLlama-70B-Instruct Gemini-Pro
Builds on: HumanEval+ CRUXEval CodeT AlphaCodium
Closest papers by meaning
- DyCodeEval: Dynamic Benchmarking of Reasoning Capabilities in Code Large Language Models Under Data Contamination (ICML 2025 · similarity 0.92)
- EvoCodeBench: An Evolving Code Generation Benchmark with Domain-Specific Evaluations (NeurIPS 2024 · similarity 0.91)
- How efficient is LLM-generated code? A rigorous & high-standard benchmark (ICLR 2025 · similarity 0.90)
- DOMAINEVAL: An Auto-Constructed Benchmark for Multi-Domain Code Generation (AAAI 2025 · similarity 0.90)
- Unsupervised Evaluation of Code LLMs with Round-Trip Correctness (ICML 2024 · similarity 0.89)
- Mercury: A Code Efficiency Benchmark for Code Large Language Models (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.