AI Paper Atlas

Research map / Large language models / Language, safety and interpretability

SpotIt: Evaluating Text-to-SQL Evaluation with Formal Verification

Rocky Klopfenstein, YANG HE, Andrew Tremante, Yuepeng Wang, Nina Narodytska, Haoze Wu

ICLR 2026 · Large language models › Language, safety and interpretability · approach: Data and evaluation methodology · 5 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
add date and string SMT encodings to the verifier (ablation)SpotIt (VeriEQL) · BIRD-dev, CSC-SQL + XiYanSQLverifier coverage (%): 84.83 -> 94.88Improved
add date and string SMT encodings to the verifier (ablation)SpotIt (VeriEQL) · BIRD-dev, Alpha-SQLverifier coverage (%): 84.87 -> 93.89Improved
add date and string SMT encodings to the verifier (ablation)SpotIt (VeriEQL) · BIRD-dev, CHESSverifier coverage (%): 87.40 -> 97.13Improved
replace test-based EX with SpotIt verificationCSC-SQL + XiYanSQL · BIRD-devaccuracy (%): 71.32 -> 58.80Hurt
add cross-checking of counterexamples across frameworks (ablation)CSC-SQL + XiYanSQL · BIRD-devaccuracy (%): 58.80 -> 57.82Hurt
replace test-based EX with SpotIt verificationGenaSQL-2 · BIRD-devaccuracy (%): 70.53 -> 59.84Hurt
replace test-based EX with SpotIt verificationAlpha-SQL · BIRD-devaccuracy (%): 69.36 -> 55.87Hurt
replace test-based EX with SpotIt verificationSLM-SQL · BIRD-devaccuracy (%): 63.43 -> 51.37Hurt
replace test-based EX with SpotIt verificationOmniSQL · Spider 2.0 (135 SQLite questions)accuracy (%): 34.1 -> 22.2Hurt
replace test-based EX with SpotIt verificationGPT-5 · Spider 2.0 (135 SQLite questions)accuracy (%): 42.2 -> 36.3Hurt
increase verification bound K from 1 to 2, then beyond 3 (ablation)SpotIt on CSC-SQL + XiYanSQL · BIRD-devdifferentiating databases found: significantly more from K=1 to 2, marginal past K=3Mixed

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy