Research map / Large language models / Agents, code and math
BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces BigCodeBench, a 1,140-task Python benchmark (plus NL-oriented BigCodeBench-Instruct) requiring multi-library function calls and complex instruction following, built via human-LLM collaboration, and evaluates 60 LLMs.
Problem
Existing code benchmarks like HumanEval and MBPP are short, self-contained, and saturated, and do not test diverse multi-library function-call use or complex instruction following in realistic programming tasks.
Approach
Source realistic tasks from Stack Overflow API seed examples with GPT-4, refine programs and write rigorous unittest test suites through GPT-4 Code Interpreter under human supervision, then derive a natural-language-oriented instruct variant by rule-based docstring transformation.
Key results
- Best model GPT-4o solves about 60% of BigCodeBench-Complete tasks and under 50% of BigCodeBench-Instruct, versus 97% human performance.
- Pearson correlation of model ranks between BigCodeBench-Complete and BigCodeBench-Instruct is 0.982.
- Instruction-tuned LLMs average about 8.5% lower Pass@1 on BigCodeBench-Instruct than on BigCodeBench-Complete.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply instruction tuning | paired base LLMs · BigCodeBench-Complete | calibrated Pass@1: 35.7% -> 40.7% | Helped |
| use less verbose, natural-language-oriented prompts | LLMs · BigCodeBench-Instruct vs BigCodeBench-Complete | Pass@1: average decrease of 8.5% | Hurt |
| add missing setup, such as import statements and global constants, to generations | instruction-tuned LLM generations · BigCodeBench-Complete | Pass@1 | Mixed |
| use the CodeQwen1.5-7B instruction-tuned model instead of its base model | CodeQwen1.5-7B · BigCodeBench-Complete | task performance | No clear effect |
| evaluate on the shorter Instruct prompts rather than the Complete prompts | instruction-tuned LLMs · BigCodeBench-Instruct vs BigCodeBench-Complete | Pass@1: average decrease of 8.5% | Hurt |
Surprising findings
- Some instruction-tuned models such as GPT-4 often omit essential import statements from long prompts, causing test failures (model laziness).
- Models frequently use different function calls than the ground truth, and using more diverse function calls is more likely to cause failures.
Limitations
- Python-only; not easily extended to other languages.
- Flaky tests cause roughly 0.6% uncontrollable Pass@1 variation; some tests rely on network access.
- Test suites do not guarantee correctness of all LLM-generated solutions that use different function calls.
Experimental setup
Datasets: ODEX Stack Overflow archive (StarCoder2 preprocessing) StarCoderData
Benchmarks: BigCodeBench-Complete BigCodeBench-Instruct BigCodeBench-Hard
Metrics: Pass@1 Pass@5 calibrated Pass@1 branch coverage
Baselines: HumanEval MBPP DS-1000 ODEX HumanEvalPack SWE-bench
Builds on: ODEX HumanEval MBPP EvalPlus MixEval
Closest papers by meaning
- SemCoder: Training Code Language Models with Comprehensive Semantics Reasoning (NeurIPS 2024 · similarity 0.90)
- The Berkeley Function Calling Leaderboard (BFCL): From Tool Use to Agentic Evaluation of Large Language Models (ICML 2025 · similarity 0.90)
- ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs (ICLR 2024 · similarity 0.89)
- CRAFT: Customizing LLMs by Creating and Retrieving from Specialized Toolsets (ICLR 2024 · similarity 0.89)
- EffiCoder: Enhancing Code Generation in Large Language Models through Efficiency-Aware Fine-tuning (ICML 2025 · similarity 0.89)
- CodeMMLU: A Multi-Task Benchmark for Assessing Code Understanding & Reasoning Capabilities of CodeLLMs (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.