Research map / Multi-agent systems and games / LLM agents

SciAgentGym: Benchmarking Multi-Step Scientific Tool-Use in LLM Agents

ICML 2026 · Multi-agent systems and games › LLM agents · approach: Robust and efficient agent execution

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces SciAgentGym, an interactive environment with 1,780 scientific tools, the SciAgentBench evaluation (259 tasks), and SciForge, a dependency-graph data synthesis method used to fine-tune Qwen3-VL-8B/4B into SciAgent-8B/4B.

Problem

Existing scientific benchmarks are static QA and general tool-use benchmarks lack domain-specific scientific tools, so multi-step, tool-mediated scientific workflows of LLM agents are not evaluated or trained for.

Approach

A typed scientific tool environment with execution feedback, a tiered benchmark (L1-L3) with golden traces, and SciForge which samples executable tool dependency graphs, executes them to get verified traces (including error-recovery trajectories), and converts traces into natural-language questions for SFT.

Key results

• GPT-5 achieves 41.3% overall success rate with tools on SciAgentBench, dropping from 58.8% on L1 to 34.6% on L3.
• SciAgent-8B improves +6.7 points with tools (23.4 to 30.1 SR), outperforming Qwen3-VL-235B-Inst (23.9%) per the paper's claims.
• Models average 47.4% on L1 but 16.4% on L3 tasks, a 65.4% relative drop.

Surprising findings

• Other-domain (non-scientific) tool fine-tuning causes a 4.9-point drop, indicating negative transfer.
• Tool-call frequency is negatively correlated with success (r = -0.18), as weaker models fall into repetitive invocation loops.

Limitations

• SPL measures alignment with a single expert reference path rather than optimality of planning.
• Benchmark tasks are filtered to be hard for frontier LLMs (below 50% accuracy), which may bias difficulty.
• Life Sciences and Materials subsets are small (32 and 37 tasks).

Experimental setup

Datasets: ScienceQA GPQA R-Bench-V BMMR SFE SciInstruct

Benchmarks: SciAgentBench

Metrics: success rate SPL

Baselines: GPT-5 GPT-4o Claude-Sonnet-4 Gemini-2.5-Pro Qwen3-VL-235B-Inst O3 O4-mini GLM-4.6V

Builds on: ReAct ToolLLM AgentGym SPL

Closest papers by meaning

• InnovatorBench: Evaluating Agents’ Ability to Conduct Innovative AI Research (ICLR 2026 · similarity 0.90)
• ScienceBoard: Evaluating Multimodal Autonomous Agents in Realistic Scientific Workflows (ICLR 2026 · similarity 0.90)
• AstaBench: Rigorous Benchmarking of AI Agents with a Scientific Research Suite (ICLR 2026 · similarity 0.89)
• UltraHorizon: Benchmarking LLM-Agent Capabilities in Ultra Long-Horizon Scenarios (ICML 2026 · similarity 0.89)
• PhysGym: Benchmarking LLMs in Interactive Physics Discovery with Controlled Priors (NeurIPS 2025 · similarity 0.89)
• AgentSwift: Efficient LLM Agent Design via Value-Guided Hierarchical Search (AAAI 2026 · similarity 0.88)