Research map / Large language models / Agents, code and math
ScienceAgentBench: Toward Rigorous Assessment of Language Agents for Data-Driven Scientific Discovery
Ziru Chen, Shijie Chen, Yuting Ning, Qianheng Zhang, Boshi Wang, Botao Yu, Yifei Li, Zeyi Liao, Chen Wei, Zitong Lu, Vishal Dey, Mingyi Xue and 8 more
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
placeholder
Problem
placeholder
Approach
placeholder
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Self-Debug instead of OpenHands CodeAct | Claude-3.5-Sonnet · ScienceAgentBench, without expert knowledge | SR: 21.6 -> 32.4 | Improved |
| Use Self-Debug instead of direct prompting | Claude-3.5-Sonnet · ScienceAgentBench, without expert knowledge | SR: 17.7 -> 32.4 | Improved |
| Use Self-Debug instead of direct prompting | Claude-3.5-Sonnet · ScienceAgentBench, with expert knowledge | SR: 21.6 -> 34.3 | Improved |
| Use Self-Debug instead of OpenHands CodeAct | Claude-3.5-Sonnet · ScienceAgentBench, without expert knowledge | API Cost: $0.958 -> $0.057 | Improved |
| Add expert-provided knowledge | Claude-3.5-Sonnet + Self-Debug · ScienceAgentBench | SR: 32.4 -> 34.3 | Improved |
| Add expert-provided knowledge | Claude-3.5-Sonnet + Self-Debug · ScienceAgentBench | VER: 92.2 -> 86.3 | Hurt |
| Use Self-Debug instead of direct prompting | Claude-3.5-Sonnet · ScienceAgentBench, with expert knowledge | VER: 41.2 -> 86.3 | Improved |
| Use OpenAI o1-preview with Self-Debug | OpenAI o1-preview · ScienceAgentBench, without expert knowledge | SR: 34.3 -> 42.2 | Improved |
Closest papers by meaning
- ScienceBoard: Evaluating Multimodal Autonomous Agents in Realistic Scientific Workflows (ICLR 2026 · similarity 0.92)
- DiscoveryBench: Towards Data-Driven Discovery with Large Language Models (ICLR 2025 · similarity 0.91)
- SciAgentGym: Benchmarking Multi-Step Scientific Tool-Use in LLM Agents (ICML 2026 · similarity 0.90)
- AstaBench: Rigorous Benchmarking of AI Agents with a Scientific Research Suite (ICLR 2026 · similarity 0.89)
- SciCode: A Research Coding Benchmark Curated by Scientists (NeurIPS 2024 · similarity 0.89)
- SciResearchBench: Benchmarking AI Agents on Complex Scientific Literature Discovery (NeurIPS 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
