Research map / Large language models / Agents, code and math
AutoCodeBench: Large Language Models are Automatic Code Benchmark Generators
Changzhi Zhou, Ao Liu, Yuchi Deng, Zhiying Zeng, Tao Zhang, Haotian Zhu, Jianwei Cai, Yue Mao, Chenchen Zhang, Lingyun Tan, ZiyanXU, Bohui Zhai and 4 more
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use reasoning mode instead of non-reasoning mode (ablation) | Claude Opus 4 · AutoCodeBench | Pass@1 average (%): 50.9 -> 52.4 | Improved |
| use reasoning mode instead of non-reasoning mode (ablation) | Qwen3-32B · AutoCodeBench | Pass@1 average (%): 31.0 -> 41.7 | Improved |
| add up to three turns of sandbox execution-error feedback refinement (ablation) | DeepSeek-V3-0324 · AutoCodeBench | Pass@1 average (%): 48.1 -> 59.7 | Improved |
| add up to three turns of sandbox execution-error feedback refinement (ablation) | Qwen2.5-Coder-32B-Instruct · AutoCodeBench | Pass@1 average (%): 35.8 -> 47.4 | Improved |
| add sandbox feedback refinement (ablation) | Qwen3-8B · AutoCodeBench | Pass@1 average (%): 23.3 -> 30.2 | Improved |
| filter out easy problems using DeepSeek-Coder-V2-Lite (10 samples) (ablation) | AutoCodeGen data pool · AutoCodeBench subset (6 languages) | Pass@1 average (%): Gemini2.5 Pro 51.4 -> 31.6; Qwen2.5-Coder-32B-Instruct 39.9 -> 17.1 | Improved |
| add LLM-as-Critic filtering after difficulty filtering (ablation) | AutoCodeGen data pool · AutoCodeBench subset (6 languages) | Pass@1 average (%): Gemini2.5 Pro 31.6 -> 38.7; DeepSeek-V3-0324 25.7 -> 31.6 | Improved |
| evaluate on multi-logical problems instead of full dataset (ablation) | all evaluated models · AutoCodeBench (multi-logic subset, 1,622 problems) | Pass@1 | Hurt |
| evaluate on low-resource languages versus popular languages (ablation) | top models · AutoCodeBench | Pass@1 spread across model families: popular 50.4-53.8; low-resource 45.3-62.0 | Mixed |
| scale up model size within Qwen3 thinking series (ablation) | Qwen3 (Think) series · AutoCodeBench | Pass@1 average (%): Qwen3-1.7B 11.2 -> Qwen3-8B 28.5 -> Qwen3-32B 41.7 | Improved |
Closest papers by meaning
- BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions (ICLR 2025 · similarity 0.92)
- McEval: Massively Multilingual Code Evaluation (ICLR 2025 · similarity 0.92)
- Teaching Large Language Models to Self-Debug (ICLR 2024 · similarity 0.91)
- How efficient is LLM-generated code? A rigorous & high-standard benchmark (ICLR 2025 · similarity 0.91)
- EvoCodeBench: An Evolving Code Generation Benchmark with Domain-Specific Evaluations (NeurIPS 2024 · similarity 0.91)
- EffiCoder: Enhancing Code Generation in Large Language Models through Efficiency-Aware Fine-tuning (ICML 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
