Research map / Large language models / Agents, code and math
Beyond Problem Solving: UOJ-Bench for Evaluating Code Generation, Hacking, and Repair in Competitive Programming
Tingqiang Xu, Hangrui Zhou, Tianle Cai, Alex Gu, Kaifeng Lyu
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| sample many candidates (Pass@k) instead of one | Gemini-3-pro-preview · Hacking (Hard) 100-sample subset | success rate: 93 out of 100 hacks found | Improved |
| sample many candidates (Pass@k) instead of one | Gemini-3-pro-preview · Repair (Hard) 50-sample subset | success rate: 41 out of 50 repaired | Improved |
| use agentic ReAct feedback for hacking instead of parallel sampling | GPT-OSS-120B · Hacking (Hard) | success rate over 10 turns | No clear effect |
| use agentic iterative debugging for repair instead of direct sampling | GPT-OSS-120B · Repair (Hard) | success rate over 10 turns | Improved |
| hard (covert error) instead of easy (overt error) hacking | GPT-5.2-high · UOJ-Bench hacking | Pass@1: 71.61 -> 58.60 | Hurt |
| hard instead of easy repair | GPT-OSS-120B · UOJ-Bench repair | Pass@1: 20.20 -> 9.72 | Hurt |
| problem difficulty from Easy to Ultrahard | GPT-5.2-high · UOJ-Bench generation | Pass@1: 88.89 -> 27.33 | Hurt |
| evaluate on more recent problems | LLMs · Hard problems generation | Pass@1 | Hurt |
| evaluate on more recent problems | LLMs · Hard problems hacking and repair | Pass@1 | No clear effect |
| run GPT-OSS-120B hacking with test-time scaling on zero-day submissions | GPT-OSS-120B · Zero-Day-Hacking-5K | fraction of submissions with new errors: over 10% | Improved |
Closest papers by meaning
- LiveCodeBench Pro: How Do Olympiad Medalists Judge LLMs in Competitive Programming? (NeurIPS 2025 · similarity 0.90)
- DyCodeEval: Dynamic Benchmarking of Reasoning Capabilities in Code Large Language Models Under Data Contamination (ICML 2025 · similarity 0.89)
- LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code (ICLR 2025 · similarity 0.88)
- CS-Bench: A Comprehensive Benchmark for Large Language Models towards Computer Science Mastery (ICLR 2025 · similarity 0.88)
- Rethinking Verification for LLM Code Generation: From Generation to Testing (NeurIPS 2025 · similarity 0.88)
- ICPC-Eval: Probing the Frontiers of LLM Reasoning with Competitive Programming Contests (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
