Research map / Multi-agent systems and games / LLM agents
Spider2-V: How Far Are Multimodal Agents From Automating Data Science and Engineering Workflows?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Spider2-V, a 494-task multimodal agent benchmark for professional data science and engineering workflows in executable Ubuntu VMs, where the best VLM (GPT-4V) reaches 14.0% success.
Problem
Existing agent benchmarks cover either code-only data tasks or generic GUI workflows; there is no executable benchmark testing whole data science and engineering pipelines that mix code generation with GUI control in enterprise data software.
Approach
Build an OSWorld-based executable environment with 20 professional data tools (including cloud services requiring real accounts), 494 tutorial-derived tasks with automatic setup and execution-based evaluation, and a retrieval document warehouse of official docs for RAG; baseline agents use Set-of-Mark, execution feedback and RAG.
Key results
- GPT-4V achieves the best overall success rate of 14.0% on Spider2-V, with GPT-4o at 13.8%.
- Success drops to 1.2% on the hardest tasks (more than 15 action steps) for GPT-4o with SoM+EF+RAG.
- Even with verbose step-by-step instructions, GPT-4o reaches only 16.2% success; tasks requiring authentic cloud accounts reach 10.6%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| provide verbose step-by-step instructions instead of abstract instructions | GPT-4o SoM+EF+RAG · Spider2-V | success rate (%): 11.3 -> 16.2 | Helped |
| use pyautogui code instead of JSON dict actions (ablation) | GPT-4o · Spider2-V task subset with a11ytree observations | success rate (%): 10.5 -> 12.6 | Helped |
| use a11ytree observations instead of screenshot-only observations (ablation) | GPT-4o with pyautogui actions · Spider2-V task subset | success rate (%): 4.2 -> 12.6 | Helped |
| add Set-of-Mark to align screenshots and a11ytree (ablation) | GPT-4o with pyautogui actions · Spider2-V task subset | success rate (%): 11.4 -> 15.6 | Helped |
| add execution feedback (ablation) | GPT-4o with pyautogui actions · Spider2-V task subset | success rate (%): 11.4 -> 13.6 | Helped |
| add retrieval augmentation (ablation) | GPT-4o with pyautogui actions · Spider2-V task subset | success rate (%): 11.4 -> 14.4 | Helped |
| combine Set-of-Mark, execution feedback, and retrieval augmentation (ablation) | GPT-4o with pyautogui actions · Spider2-V task subset | success rate (%): 11.4 -> 16.3 | Helped |
| use pure text rather than Markdown Syntax for retrieved documents (ablation) | retrieval-augmented agent · Spider2-V 130-task subset | success rate (%): 15.38 -> 16.92 | Helped |
| use pure text rather than Simplified HTML for retrieved documents (ablation) | retrieval-augmented agent · Spider2-V 130-task subset | success rate (%): 15.38 -> 16.92 | Helped |
| increase the history window size from 0 to 3 (ablation) | GPT-4o agent · Spider2-V task subset | success rate | Helped |
| increase task difficulty from easy to hard by increasing instructed action steps | GPT-4o SoM+EF+RAG · Spider2-V | success rate (%): 38.8 -> 9.7 -> 1.2 | Hurt |
| require authentic user accounts and cloud-hosted workspaces | GPT-4o SoM+EF+RAG · Spider2-V | success rate (%): 15.6 -> 10.6 | Hurt |
Surprising findings
- Open-source LLMs (Mixtral-8x7B, Llama-3-70B) with text-only a11ytree score below 2%, far below closed-source VLMs.
- Adding a screenshot without Set-of-Mark alignment performs worse than a11ytree alone (11.4% vs 12.6% range in ablation, a screenshot-only setting only 4.2%).
Limitations
- Overall success rates remain very low (best 14.0%), so the benchmark is far from solved by current agents.
- Tasks requiring cloud accounts (BigQuery, Snowflake) suffer from network delay and pop-ups, lowering performance.
- Limited interaction history (window 3) due to context length and cost constraints.
Experimental setup
Datasets: Spider2-V tasks official software documentation (11,231 filtered pages)
Benchmarks: Spider2-V
Metrics: success rate
Baselines: Mixtral-8x7B Llama-3-70B Qwen-Max Claude-3-Opus Gemini-Pro-1.5 GPT-4o GPT-4V
Builds on: OSWorld Set-of-Mark WebArena LlamaIndex
Closest papers by meaning
- ScienceBoard: Evaluating Multimodal Autonomous Agents in Realistic Scientific Workflows (ICLR 2026 · similarity 0.87)
- Spider 2.0: Evaluating Language Models on Real-World Enterprise Text-to-SQL Workflows (ICLR 2025 · similarity 0.87)
- Windows Agent Arena: Evaluating Multi-Modal OS Agents at Scale (ICML 2025 · similarity 0.87)
- DSBench: How Far Are Data Science Agents from Becoming Data Science Experts? (ICLR 2025 · similarity 0.87)
- OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments (NeurIPS 2024 · similarity 0.86)
- Multi-modal Agent Tuning: Building a VLM-Driven Agent for Efficient Tool Usage (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.