Research map / Large language models / Language, safety and interpretability
Spider 2.0: Evaluating Language Models on Real-World Enterprise Text-to-SQL Workflows
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Spider 2.0, a benchmark of 632 enterprise-scale text-to-SQL workflow tasks (plus Spider 2.0-lite/snow text-to-SQL variants) in real codebases and cloud/local databases, showing frontier LLM code agents solve only ~21% of tasks.
Problem
Existing text-to-SQL benchmarks use small, simple databases and single SQL queries, failing to reflect real enterprise data workflows that involve large schemas, multiple SQL dialects, project codebases, documentation, and multi-step data transformation and analytics.
Approach
An agentic benchmark where the model must navigate project codebases, documentation, and live cloud/local database interfaces to iteratively write SQL/Python and produce a final result, paired with text-to-SQL variants (Spider 2.0-lite/snow) and a ReAct-style code agent (Spider-Agent) baseline.
Key results
- o1-preview with Spider-Agent achieves 21.36% success rate on Spider 2.0, versus 91.2% on Spider 1.0 and 73.0% on BIRD for GPT-4-based methods.
- On Spider 2.0-lite, the best text-to-SQL method DAIL-SQL + GPT-4o reaches only 5.68% execution accuracy.
- o1-preview achieves 23.22% overall EX on Spider 2.0-lite and 23.77% on Spider 2.0-snow with Spider-Agent.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Spider-Agent with o1-preview instead of GPT-4o code-agent frameworks | Spider 2.0 code-agent task · Spider 2.0 | success rate: 12.34% -> 21.36% | Helped |
| Use o3-mini instead of o1-preview in Spider-Agent | Spider-Agent · Spider 2.0-Lite | execution accuracy: 23.22% -> 23.40% | Helped |
| Use o1-preview instead of o3-mini in Spider-Agent | Spider-Agent · Spider 2.0-Snow | execution accuracy: 19.20% -> 23.77% | Helped |
| Use DAIL-SQL + GPT-4o instead of DIN-SQL + GPT-4o | text-to-SQL baseline · Spider 2.0-Lite | execution accuracy: 1.46% -> 5.68% | Helped |
| Use DAIL-SQL + GPT-4o instead of CHESS + GPT-4o | text-to-SQL baseline · Spider 2.0-Snow | execution accuracy: 1.28% -> 2.20% | Helped |
| Add oracle SQL function documentation (ablation) | DAIL-SQL + GPT-4o · Spider 2.0-Lite | execution accuracy: 5.68% -> 5.85% | Helped |
| Add oracle SQL function documentation (ablation) | DAIL-SQL + o1-preview · Spider 2.0-Lite | execution accuracy: 12.60% -> 9.51% | Hurt |
| Use 1-shot prompting instead of 0-shot prompting (ablation) | DAIL-SQL + GPT-4o · Spider 2.0-Lite | execution accuracy: 5.68% -> 6.40% | Helped |
| Use 3-shot prompting instead of 0-shot prompting (ablation) | DAIL-SQL + GPT-4o · Spider 2.0-Lite | execution accuracy: 5.68% -> 6.76% | Helped |
| Require handling nested columns | Spider-Agent on non-DBT projects · Spider 2.0 tasks with nested columns | success rate: 27.38% -> 10.34% | Hurt |
| Require consulting external documents | Spider-Agent on non-DBT projects · Spider 2.0 tasks with external documents | success rate: 26.64% -> 11.54% | Hurt |
| Require completing DBT project workflows | Spider-Agent · Spider 2.0 DBT-project tasks | success rate: 23.22% -> 12.82% | Hurt |
Surprising findings
- Providing oracle SQL function documentation yields only a slight improvement, and few-shot prompting has marginal impact on performance.
- SFT CodeS-15B reaches only 0.73% EX on Spider 2.0-lite despite extensive fine-tuning on text-to-SQL corpora.
Limitations
- Success rate on the full agentic benchmark is low, so the benchmark may be too hard to show incremental progress.
- Evaluation scripts use focused column matching which may produce some false negatives or positives.
- Difficulty is defined only by gold SQL token length, a single coarse metric.
Experimental setup
Datasets: Spider 1.0 BIRD
Benchmarks: Spider 2.0 Spider 2.0-lite Spider 2.0-snow Spider 1.0 BIRD
Metrics: success rate execution accuracy
Baselines: Reflexion CodeR AutoEval DIN-SQL DAIL-SQL CHESS SFT CodeS-15B
Builds on: ReAct Reflexion InterCode DAIL-SQL CHESS
Closest papers by meaning
- Alpha-SQL: Zero-Shot Text-to-SQL using Monte Carlo Tree Search (ICML 2025 · similarity 0.87)
- LogicCat: A Chain-of-Thought Text-to-SQL Benchmark for Complex Reasoning (AAAI 2026 · similarity 0.87)
- PARROT: A Benchmark for Evaluating LLMs in Cross-System SQL Translation (NeurIPS 2025 · similarity 0.87)
- DAComp: Benchmarking Data Agents across the Full Data Intelligence Lifecycle (ICLR 2026 · similarity 0.87)
- Spider2-V: How Far Are Multimodal Agents From Automating Data Science and Engineering Workflows? (NeurIPS 2024 · similarity 0.87)
- Structure-Guided Large Language Models for Text-to-SQL Generation (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.