Research map / Large language models / Agents, code and math
Code2Bench: Scaling Source and Rigor for Dynamic Benchmark Construction
Zhe Zhang, Runlin Liu, Aishan Liu, Xingyu Liu, Xiang Gao, Hailong Sun
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| evaluate on fresh real-world SC-Python tasks instead of HumanEval | Claude-4-sonnet · SC-Python vs HumanEval | Pass@1 (%): 97 -> 40.1 | Hurt |
| switch from SC-Python to WSC-Python tasks | LLMs · Code2Bench-2509 | Pass@1 (%): Llama-4-scout 25.8 -> 32.5; Claude-4-sonnet 40.1 -> 38.7 | Mixed |
| switch from Python SC to Java SC tasks | LLMs · Code2Bench-2509 | Pass@1 (%): Claude-4-sonnet 40.1 -> 47.4; DeepSeek-V3 34.4 -> 47.8 | Improved |
| scale tests with PBT (about 500 tests) rather than sparse tests (ablation) | Code2Bench · SC-Python | near-perfect failure rate (%): 6.94% of submissions are near-perfect failures | Improved |
Closest papers by meaning
- LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code (ICLR 2025 · similarity 0.90)
- CoRe: Benchmarking LLMs’ Code Reasoning Capabilities through Static Analysis Tasks (NeurIPS 2025 · similarity 0.90)
- How efficient is LLM-generated code? A rigorous & high-standard benchmark (ICLR 2025 · similarity 0.90)
- SWT-Bench: Testing and Validating Real-World Bug-Fixes with Code Agents (NeurIPS 2024 · similarity 0.90)
- DyCodeEval: Dynamic Benchmarking of Reasoning Capabilities in Code Large Language Models Under Data Contamination (ICML 2025 · similarity 0.89)
- BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
