Research map / Large language models / Language, safety and interpretability
SpotIt: Evaluating Text-to-SQL Evaluation with Formal Verification
Rocky Klopfenstein, YANG HE, Andrew Tremante, Yuepeng Wang, Nina Narodytska, Haoze Wu
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add date and string SMT encodings to the verifier (ablation) | SpotIt (VeriEQL) · BIRD-dev, CSC-SQL + XiYanSQL | verifier coverage (%): 84.83 -> 94.88 | Improved |
| add date and string SMT encodings to the verifier (ablation) | SpotIt (VeriEQL) · BIRD-dev, Alpha-SQL | verifier coverage (%): 84.87 -> 93.89 | Improved |
| add date and string SMT encodings to the verifier (ablation) | SpotIt (VeriEQL) · BIRD-dev, CHESS | verifier coverage (%): 87.40 -> 97.13 | Improved |
| replace test-based EX with SpotIt verification | CSC-SQL + XiYanSQL · BIRD-dev | accuracy (%): 71.32 -> 58.80 | Hurt |
| add cross-checking of counterexamples across frameworks (ablation) | CSC-SQL + XiYanSQL · BIRD-dev | accuracy (%): 58.80 -> 57.82 | Hurt |
| replace test-based EX with SpotIt verification | GenaSQL-2 · BIRD-dev | accuracy (%): 70.53 -> 59.84 | Hurt |
| replace test-based EX with SpotIt verification | Alpha-SQL · BIRD-dev | accuracy (%): 69.36 -> 55.87 | Hurt |
| replace test-based EX with SpotIt verification | SLM-SQL · BIRD-dev | accuracy (%): 63.43 -> 51.37 | Hurt |
| replace test-based EX with SpotIt verification | OmniSQL · Spider 2.0 (135 SQLite questions) | accuracy (%): 34.1 -> 22.2 | Hurt |
| replace test-based EX with SpotIt verification | GPT-5 · Spider 2.0 (135 SQLite questions) | accuracy (%): 42.2 -> 36.3 | Hurt |
| increase verification bound K from 1 to 2, then beyond 3 (ablation) | SpotIt on CSC-SQL + XiYanSQL · BIRD-dev | differentiating databases found: significantly more from K=1 to 2, marginal past K=3 | Mixed |
Closest papers by meaning
- SWE-SQL: Illuminating LLM Pathways to Solve User SQL Issues in Real-World Applications (NeurIPS 2025 · similarity 0.88)
- SQLens: An End-to-End Framework for Error Detection and Correction in Text-to-SQL (NeurIPS 2025 · similarity 0.87)
- Spider 2.0: Evaluating Language Models on Real-World Enterprise Text-to-SQL Workflows (ICLR 2025 · similarity 0.87)
- LogicCat: A Chain-of-Thought Text-to-SQL Benchmark for Complex Reasoning (AAAI 2026 · similarity 0.86)
- PARROT: A Benchmark for Evaluating LLMs in Cross-System SQL Translation (NeurIPS 2025 · similarity 0.86)
- SQLFixAgent: Towards Semantic-Accurate Text-to-SQL Parsing via Consistency-Enhanced Multi-Agent Collaboration (AAAI 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
