Research map / Large language models / Agents, code and math
AutoCode: LLMs as Problem Setters for Competitive Programming
Shang Zhou, Zihan Zheng, Kaiyuan Liu, Zeyu Shen, Zerui Cheng, Zexing Chen, Hansen He, Jianzhu Yao, Huanzhi Mao, Qiuyang Mang, Tianfu Fu, Beichen Li and 9 more
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add prompt optimization (ablation) | AutoCode · 720 Codeforces problems | FNR (%): 2.9 -> 1.2 | Improved |
| add exhaustive small-data generator strategy (ablation) | AutoCode · 720 Codeforces problems | FPR (%): 1.7 -> 1.3 | Improved |
| add random/extreme generator strategy (ablation) | AutoCode · 720 Codeforces problems | FPR (%): 1.6 -> 1.3 | Improved |
| add TLE-inducing generator strategy (ablation) | AutoCode · 720 Codeforces problems | consistency (%): 98.6 -> 98.7 | No clear effect |
| use AutoCode test generation instead of HardTests tests | test suites · 7538-problem benchmark | consistency (%): 81.0 -> 91.1 | Improved |
| filter generated problems with brute-force dual verification (ablation) | problem generation · generated problems | reference correctness (%): 86 -> 94 | Improved |
| use LLM (o3) judgments instead of human experts for problem quality | problem quality assessment · generated problems | correlation: 0.07 (quality), 0.11 (novelty) | Hurt |
Closest papers by meaning
- LiveCodeBench Pro: How Do Olympiad Medalists Judge LLMs in Competitive Programming? (NeurIPS 2025 · similarity 0.90)
- ICPC-Eval: Probing the Frontiers of LLM Reasoning with Competitive Programming Contests (NeurIPS 2025 · similarity 0.89)
- CPRet: A Dataset, Benchmark, and Model for Retrieval in Competitive Programming (NeurIPS 2025 · similarity 0.88)
- Rethinking Verification for LLM Code Generation: From Generation to Testing (NeurIPS 2025 · similarity 0.87)
- rStar-Coder: Scaling Competitive Code Reasoning with a Large-Scale Verified Dataset (NeurIPS 2025 · similarity 0.87)
- LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
