Research map / Large language models / Language, safety and interpretability
Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLs
Jinyang Li, Binyuan Hui, GE QU, Jiaxi Yang, Binhua Li, Bowen Li, Bailin Wang, Bowen Qin, Ruiying Geng, Nan Huo, Xuanhe Zhou, Chenhao Ma and 5 more
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add external knowledge evidence to prompt (ablation) | GPT-4 · BIRD test EX | execution accuracy: 34.88 -> 54.89 | Improved |
| add external knowledge evidence to prompt (ablation) | ChatGPT · BIRD dev EX | execution accuracy: 24.05 -> 37.22 | Improved |
| add external knowledge evidence to prompt (ablation) | T5-3B · BIRD dev EX | execution accuracy: 10.37 -> 23.34 | Improved |
| add chain-of-thought prompting | ChatGPT · BIRD dev EX w/o knowledge | execution accuracy: 24.05 -> 25.88 | Improved |
| add chain-of-thought prompting on top of knowledge evidence | ChatGPT · BIRD dev EX w/ knowledge | execution accuracy: 37.22 -> 36.64 | Hurt |
| add DIN-SQL prompting (value sampling, few-shot, self-correction) | GPT-4 · BIRD test EX w/ knowledge | execution accuracy: 54.89 -> 55.90 | Improved |
| add DIN-SQL prompting | GPT-4 · BIRD test VES w/ knowledge | VES: 60.77 -> 59.44 | Hurt |
| two-stage optimization: expert rewrite of correct SQLs | ChatGPT SQLs · 10 dev examples | time saving: 77.75% | Improved |
| configure indexes in the database | SQL queries · BIRD efficiency analysis | time saving: 87.3% | Improved |
| scale model from T5-Base to T5-3B (w/ knowledge) | T5 · BIRD dev EX | execution accuracy: 11.54 -> 23.34 | Improved |
Closest papers by meaning
- Structure-Guided Large Language Models for Text-to-SQL Generation (ICML 2025 · similarity 0.91)
- PARROT: A Benchmark for Evaluating LLMs in Cross-System SQL Translation (NeurIPS 2025 · similarity 0.90)
- CogSQL: A Cognitive Framework for Enhancing Large Language Models in Text-to-SQL Translation (AAAI 2025 · similarity 0.88)
- Spider 2.0: Evaluating Language Models on Real-World Enterprise Text-to-SQL Workflows (ICLR 2025 · similarity 0.88)
- SQLens: An End-to-End Framework for Error Detection and Correction in Text-to-SQL (NeurIPS 2025 · similarity 0.88)
- Alpha-SQL: Zero-Shot Text-to-SQL using Monte Carlo Tree Search (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
