Research map / Large language models / Agents, code and math
ICPC-Eval: Probing the Frontiers of LLM Reasoning with Competitive Programming Contests
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces ICPC-Eval, a 118-problem benchmark from 11 recent ICPC contests with LLM-generated test generators and a Refine@K metric that scores models over K attempts using execution feedback.
Problem
Existing competitive programming benchmarks (LiveCodeBench, CodeElo, USACO) are too easy for top reasoning models, lack local evaluation with special judges, and Pass@K fails to capture iterative reflection with execution feedback.
Approach
Curates top-difficulty ICPC problems with LLM-synthesized random and corner-case test generators validated against known wrong programs, and proposes Refine@K, which allows K rounds of solution repair using compile/sample/hidden-test feedback instead of i.i.d. resampling.
Key results
- o3-mini High achieves the top Refine@5 of 28.8% on ICPC-Eval, while Gemini 2.5 Pro Exp reaches 22.0% and DeepSeek R1 14.4%.
- On LiveCodeBench o3-mini High scores 67.4% vs 28.8% on ICPC-Eval, indicating the benchmark is substantially harder.
- For reasoning models QwQ-32B and DeepSeek-R1, Refine@K consistently outperforms Pass@K across K, whereas non-reasoning DeepSeek-V3 and Qwen2.5-Coder-32B show Pass@K above Refine@K.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Refine@K instead of Pass@K to evaluate reasoning models (ablation) | QwQ-32B and DeepSeek R1 · ICPC-Eval | code correctness | Helped |
| Use Refine@K instead of Pass@K to evaluate non-reasoning models (ablation) | Qwen-2.5-Coder-32B and DeepSeek V3 · ICPC-Eval | code correctness | Hurt |
| Increase the number of refinement turns with execution feedback | reasoning models · ICPC-Eval | Refine@K | Helped |
| Increase the number of refinement turns with execution feedback | non-reasoning models · ICPC-Eval | Refine@K | No clear effect |
Surprising findings
- Non-reasoning models such as DeepSeek V3 and Claude 3.5 Sonnet can be hurt by execution feedback, with Pass@K exceeding Refine@K.
- Computational geometry and search algorithm problems were unsolved by all models except o3-mini High and Gemini 2.5 Pro Exp.
Limitations
- Only 118 problems from 11 contests, limiting statistical power.
- Claude 3.7 Sonnet thinking-mode results withheld due to API performance issues.
- Test generators rely on LLM synthesis and validation against only three known incorrect programs per problem.
Experimental setup
Datasets: ICPC contest problems from QOJ and VJudge (11 contests, 118 retained problems)
Benchmarks: ICPC-Eval
Metrics: Refine@5 Pass@K
Baselines: LiveCodeBench CodeElo Pass@K Claude 3.5 Sonnet DeepSeek V3
Builds on: Pass@K LiveCodeBench CodeElo
Closest papers by meaning
- LiveCodeBench Pro: How Do Olympiad Medalists Judge LLMs in Competitive Programming? (NeurIPS 2025 · similarity 0.91)
- rStar-Coder: Scaling Competitive Code Reasoning with a Large-Scale Verified Dataset (NeurIPS 2025 · similarity 0.89)
- CS-Bench: A Comprehensive Benchmark for Large Language Models towards Computer Science Mastery (ICLR 2025 · similarity 0.88)
- ZebraLogic: On the Scaling Limits of LLMs for Logical Reasoning (ICML 2025 · similarity 0.88)
- DyCodeEval: Dynamic Benchmarking of Reasoning Capabilities in Code Large Language Models Under Data Contamination (ICML 2025 · similarity 0.88)
- CodeCrash: Exposing LLM Fragility to Misleading Natural Language in Code Reasoning (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.