Research map / Large language models / Agents, code and math
SciCode: A Research Coding Benchmark Curated by Scientists
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces SciCode, a scientist-curated benchmark of 80 research-level natural-science coding problems (338 subproblems) and evaluates ten LMs, finding the best model solves only 4.6% of main problems in the standard setting.
Problem
Existing code and reasoning benchmarks are saturated or not realistic for measuring whether LMs can write code for real scientific research problems across natural science fields.
Approach
Scientist-authored, multi-step research coding problems spanning 16 natural-science subfields, each decomposed into subproblems with gold solutions and domain-specific test cases, evaluated under toggles for background knowledge and gold vs. generated prior solutions.
Key results
- Claude3.5-Sonnet achieves the best standard-setting results: 26.0% subproblem and 4.6% main-problem pass@1.
- Providing scientist-written background raises Claude3.5-Sonnet and GPT-4o subproblem pass@1 to 35.4%, and best main-problem pass@1 to 12.3% (Claude3.5-Sonnet).
- Open models fail to solve any main problem in the standard setting (Llama-3-70B and Mixtral-8x22B at 0.0%), while Deepseek-Coder-v2 reaches 3.1%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Provide scientist-written scientific background while carrying over generated prior-subproblem solutions | Claude3.5-Sonnet · SciCode | main problem pass@1: 4.6 -> 12.3 | Helped |
| Provide scientist-written scientific background while carrying over generated prior-subproblem solutions | GPT-4o · SciCode | main problem pass@1: 1.5 -> 9.2 | Helped |
| Provide scientist-written scientific background while carrying over generated prior-subproblem solutions | Deepseek-Coder-v2 · SciCode | main problem pass@1: 3.1 -> 4.6 | Helped |
| Condition on more gold solutions from previous subproblems | the three evaluated models · SciCode subproblem evaluation | pass@1 | Mixed |
Surprising findings
- Open models solve some subproblems but no main problems, indicating error accumulation across steps is the main bottleneck.
- Performance declines when conditioning on more than 9 previous gold solutions, possibly due to long-context difficulty.
Limitations
- Small benchmark (80 main problems, 65 in test split) limits statistical power, as shown by coarse 1.5% increments.
- Problems were simplified during revisions because most were too hard for top models, and evaluation relies on scientist-authored test cases.
Experimental setup
Datasets: SciCode
Benchmarks: SciCode
Metrics: pass@1 (subproblem) pass@1 (main problem)
Baselines: GPT-4-Turbo Claude3-Opus Claude3-Sonnet Gemini 1.5 Pro Llama-3-70B-Instruct Mixtral-8x22B-Instruct Deepseek-Coder-v2 Qwen2-72B-Instruct
Builds on: HumanEval MTPB
Closest papers by meaning
- SciBench: Evaluating College-Level Scientific Problem-Solving Abilities of Large Language Models (ICML 2024 · similarity 0.90)
- SciInstruct: a Self-Reflective Instruction Annotated Dataset for Training Scientific Language Models (NeurIPS 2024 · similarity 0.90)
- AI-Researcher: Autonomous Scientific Innovation (NeurIPS 2025 · similarity 0.89)
- From Reproduction to Replication: Evaluating Research Agents with Progressive Code Masking (ICLR 2026 · similarity 0.88)
- MOOSE-Chem2: Exploring LLM Limits in Fine-Grained Scientific Hypothesis Discovery via Hierarchical Search (NeurIPS 2025 · similarity 0.88)
- ScienceBoard: Evaluating Multimodal Autonomous Agents in Realistic Scientific Workflows (ICLR 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.