Research map / Multi-agent systems and games / LLM agents
DiscoveryWorld: A Virtual Environment for Developing and Evaluating Automated Scientific Discovery Agents
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces DiscoveryWorld, a text-based virtual environment with 120 parametric scientific discovery tasks (8 themes x 3 difficulties x 5 seeds) plus 10 unit-test themes, and shows strong LLM agents (GPT-4o) struggle versus human scientists.
Problem
Evaluating AI agents' end-to-end scientific discovery (hypothesis formation, experimentation, analysis, conclusions) is hard because real-world experiments are expensive or infeasible.
Approach
A simulated, text-based (with optional 2D visual) environment with parametric discovery tasks across eight scientific themes and three difficulty levels, scored by task completion, procedural report cards, and graded explanatory knowledge.
Key results
- ReAct completed 38% of easy tasks and 18% of challenge tasks, while human scientists averaged 66% task completion across tasks.
- Hypothesizer discovered only 34% of gold knowledge on easy tasks and 8% on challenge tasks.
- Baseline agents averaged 60%+ completion on the 10 unit tests, but failed most end-to-end discovery tasks.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ReAct instead of Plan+Execute | GPT-4o discovery agents · DiscoveryWorld Easy (average) | completion: 0.18 -> 0.38 | Helped |
| Use Hypothesizer instead of ReAct | GPT-4o discovery agents · DiscoveryWorld Easy (average) | knowledge: 0.25 -> 0.34 | Helped |
| Use Plan+Execute instead of ReAct | GPT-4o discovery agents · DiscoveryWorld Normal (average) | procedure: 0.63 -> 0.64 | Helped |
| Use Hypothesizer instead of Plan+Execute | GPT-4o discovery agents · DiscoveryWorld Normal (average) | knowledge: 0.02 -> 0.19 | Helped |
| Use ReAct instead of Hypothesizer | GPT-4o discovery agents · DiscoveryWorld Challenge (average) | completion: 0.08 -> 0.18 | Helped |
| Use ReAct instead of Hypothesizer | GPT-4o discovery agents · DiscoveryWorld Challenge (average) | procedure: 0.49 -> 0.63 | Helped |
| Use Hypothesizer instead of Plan+Execute | GPT-4o agents · Unit Tests (average) | completion: 0.44 -> 0.64 | Helped |
| Evaluate agents on end-to-end discovery tasks rather than unit tests | GPT-4o baseline agents · Easy tasks versus Unit Tests | completion: ReAct: 0.60 -> 0.38; Plan+Execute: 0.44 -> 0.18; Hypothesizer: 0.64 -> 0.28 | Hurt |
| Compare expert human scientists with the strongest agent baselines | DiscoveryWorld discovery tasks · Discovery tasks (average) | completion: 0.38 -> 0.66 | Helped |
Surprising findings
- Agents performed moderately on unit tests (60%+ completion) yet poorly on discovery tasks, indicating gaps in discovery-specific competencies rather than basic interaction skills.
Limitations
- Low-fidelity simulation with abstracted action space may not transfer to real-world discovery.
- High API cost of running agents on all 120 tasks (USD $3k-$10k).
- Knowledge evaluation relies on GPT-4o grading, which may be imperfect.
Experimental setup
Benchmarks: DiscoveryWorld
Metrics: task completion procedural score explanatory knowledge score
Baselines: ReAct Plan+Execute Hypothesizer human scientists
Builds on: ReAct plan-and-execute CLIN ScienceWorld
Closest papers by meaning
- PhysGym: Benchmarking LLMs in Interactive Physics Discovery with Controlled Priors (NeurIPS 2025 · similarity 0.89)
- ScienceBoard: Evaluating Multimodal Autonomous Agents in Realistic Scientific Workflows (ICLR 2026 · similarity 0.89)
- AstaBench: Rigorous Benchmarking of AI Agents with a Scientific Research Suite (ICLR 2026 · similarity 0.89)
- Agentic AI Scientists Are Not Built For Autonomous Scientific Discovery (NeurIPS 2026 · similarity 0.88)
- AI-Researcher: Autonomous Scientific Innovation (NeurIPS 2025 · similarity 0.88)
- InnovatorBench: Evaluating Agents’ Ability to Conduct Innovative AI Research (ICLR 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.