Research map / Large language models / Agents, code and math
SWE-SQL: Illuminating LLM Pathways to Solve User SQL Issues in Real-World Applications
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces BIRD-CRITIC, a benchmark of SQL issue debugging tasks from StackOverflow, plus Six-Gym (synthetic training environment via SQL-Rewind), f-Plan Boosting trajectory augmentation, and the open-source Bird-Fixer agent fine-tuned from Qwen-2.5-Coder-14B.
Problem
LLMs are strong at text-to-SQL but have not been systematically evaluated on debugging real user SQL issues, and open-source models lack training environments for this task; proprietary-model evaluation also risks contamination and privacy concerns.
Approach
Reverse-engineer realistic SQL issues from verified solution SQLs (SQL-Rewind) to build a training gym, and boost teacher trajectory yield by extracting step-by-step functional plans from issue-solution pairs and using them to guide teacher rollouts (f-Plan Boosting); decouple thought and SQL generation at inference (GTM).
Key results
- O3-Mini achieves 38.87% SR on BIRD-CRITIC-PG and 33.33% on BIRD-CRITIC-Multi, the best among evaluated baselines.
- Bird-Fixer on Qwen-2.5-Coder-14B reaches 38.11% SR on BIRD-CRITIC-PG and 29.65% on BIRD-CRITIC-Multi, exceeding Claude-3.7-Sonnet and GPT-4.1 on PG.
- f-Plan Boosting yields 2,178 successful trajectories vs 1,254 for the vanilla baseline (73.7% more).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply Bird-Fixer fine-tuning with SQL-Act and f-Plan Boosting to Qwen-2.5-Coder-14B | Qwen-2.5-Coder-14B · BIRD-CRITIC-PG | success rate: 31.32 -> 38.11 | Helped |
| Apply Bird-Fixer fine-tuning with SQL-Act and f-Plan Boosting to Qwen-2.5-Coder-14B | Qwen-2.5-Coder-14B · BIRD-CRITIC-Multi | success rate: 24.04 -> 29.65 | Helped |
| Use Bird-Fixer instead of SQL-Act alone on the reported base models | Llama-3.1-8B, Qwen-2.5-Coder-7B, Qwen-2.5-Coder-14B, Phi-4 · BIRD-CRITIC-PG and BIRD-CRITIC-Multi | success rate: +43.34%; +42.46%; +33.84%; +20.58%; +21.68%; +23.36%; +26.23%; +20.58% | Helped |
| Remove GTM, so the fine-tuned model generates thought and SQL actions directly (ablation) | Bird-Fixer · BIRD-CRITIC-PG | success rate: 33.33% | Hurt |
| Use vanilla collection trajectories instead of f-Plan Boosting trajectories (ablation) | Bird-Fixer · BIRD-CRITIC-PG | success rate: 32.45% | Hurt |
| Add f-Plan guidance to the standard SQL-Act rollout | Gemini-2.0-Flash trajectory collection on Six-Gym · Six-Gym | successful trajectories: 1,254 -> 2,178; 73.7% | Helped |
| Combine rejection sampling with f-Plan guidance instead of rejection sampling alone | Gemini-2.0-Flash trajectory collection on Six-Gym · Six-Gym | successful trajectories: 1,910 -> 2,560 | Helped |
| Use f-Plan rather than rejection sampling | Trajectory generation on Six-Gym, by SQL complexity · Complex (5+ clauses) | success rate: 37.1% -> 66.3% | Helped |
| Use f-Plan rather than rejection sampling | Trajectory generation on Six-Gym, by SQL complexity · Simple (1-2 clauses) | success rate: 62.8% -> 70.3% | Helped |
| Use f-Plan rather than rejection sampling | Trajectory generation on Six-Gym, by SQL complexity · Medium (3-4 clauses) | success rate: 58.8% -> 69.4% | Helped |
| Use f-Plan rather than rejection sampling | Trajectory generation on Six-Gym, by SQL complexity · High Keyword Diversity (10+) | success rate: 35.6% -> 54.3% | Helped |
| Use f-Plan rather than rejection sampling | Trajectory generation on Six-Gym, by SQL complexity · Nested Operations (2+ levels) | success rate: 36.8% -> 49.2% | Helped |
Surprising findings
- Agent performance of several small models declines relative to their base under SQL-Act inference alone, suggesting long interaction histories overwhelm SLMs.
- Integrating file-saving workflow steps drops success rates from roughly 30% to 10%.
Limitations
- Evaluation is single-turn and does not model multi-turn user clarification.
- Focuses on SQL content; file/workflow integration causes large performance drops (about 30% to 10%).
- Small models can degrade under agentic inference without fine-tuning.
Experimental setup
Datasets: Six-Gym BIRD-SQL dev databases BIRD-SQL training databases StackOverflow issues
Benchmarks: BIRD-CRITIC-PG BIRD-CRITIC-Multi
Metrics: success rate
Baselines: O3-Mini GPT-4.1 Claude-3.7-Sonnet Gemini-2.0-Flash-Thinking O1-Preview DeepSeek-V3 Qwen2.5-Coder-32B Tool-Act
Builds on: ReAct LLaMA-Factory BIRD-SQL InterCode
Closest papers by meaning
- PARROT: A Benchmark for Evaluating LLMs in Cross-System SQL Translation (NeurIPS 2025 · similarity 0.87)
- SQLens: An End-to-End Framework for Error Detection and Correction in Text-to-SQL (NeurIPS 2025 · similarity 0.87)
- Spider 2.0: Evaluating Language Models on Real-World Enterprise Text-to-SQL Workflows (ICLR 2025 · similarity 0.86)
- Structure-Guided Large Language Models for Text-to-SQL Generation (ICML 2025 · similarity 0.86)
- Alpha-SQL: Zero-Shot Text-to-SQL using Monte Carlo Tree Search (ICML 2025 · similarity 0.86)
- ROUTE: Robust Multitask Tuning and Collaboration for Text-to-SQL (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.