Research map / Multi-agent systems and games / LLM agents

ScienceBoard: Evaluating Multimodal Autonomous Agents in Realistic Scientific Workflows

ICLR 2026 · Multi-agent systems and games › LLM agents · approach: LLM multi-agent collaboration and safety

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces ScienceBoard, an Ubuntu VM-based environment with six open-source scientific software packages and a human-curated benchmark of execution-validated tasks for evaluating GUI/CLI computer-using agents on scientific workflows.

Problem

Existing benchmarks do not evaluate whether computer-using agents can carry out realistic, multi-application scientific workflows through GUI and CLI interfaces, and their scientific capabilities remain unclear.

Approach

A realistic VM environment that wraps professional scientific software (ChimeraX, KAlgebra, Lean 4, Celestia, GrassGIS, TeXstudio) with injected state-exposing servers, enabling fine-grained state-based evaluation of agent task completion under dual CLI/GUI action spaces and multiple observation settings.

Key results

• GPT-5 with screenshot + a11ytree reaches 24.20% overall success, and Gemini-2.5-Pro reaches 16.98% in the same setting.
• Open-source models average below 12% overall, with several near 0% in some domains.
• Human performance averages 60.27%, far above the best agent.

Surprising findings

• Agents that excel on OSWorld do not straightforwardly dominate scientific tasks.
• Set-of-Mark sometimes hurts performance because dense scientific visuals add noise.

Limitations

• Binary success flag does not capture partial credit or process correctness.
• Evaluation limited to six domains and selected open-source software.
• Results rely on a small set of tasks per domain (about 30 or fewer in some domains), so variance may be high.

Experimental setup

Datasets: ScienceBoard benchmark tasks

Benchmarks: ScienceBoard

Metrics: success rate

Baselines: GPT-4o Claude-3.7-Sonnet Gemini-2.0-Flash Gemini-2.5-Pro o3-mini Qwen2.5-VL-72B InternVL3-78B UI-TARS-1.5-7B OS-Atlas-Pro-7B UGround-V1-7B GUI-Actor-7B GPT-5

Builds on: OSWorld Set-of-Mark VisualWebArena UFO

Closest papers by meaning

• SciAgentGym: Benchmarking Multi-Step Scientific Tool-Use in LLM Agents (ICML 2026 · similarity 0.90)
• DiscoveryWorld: A Virtual Environment for Developing and Evaluating Automated Scientific Discovery Agents (NeurIPS 2024 · similarity 0.89)
• AstaBench: Rigorous Benchmarking of AI Agents with a Scientific Research Suite (ICLR 2026 · similarity 0.89)
• AI-Researcher: Autonomous Scientific Innovation (NeurIPS 2025 · similarity 0.88)
• SciCode: A Research Coding Benchmark Curated by Scientists (NeurIPS 2024 · similarity 0.88)
• PhysGym: Benchmarking LLMs in Interactive Physics Discovery with Controlled Priors (NeurIPS 2025 · similarity 0.88)