Research map / Large language models / Language, safety and interpretability
Confidence Estimation for Error Detection in Text-to-SQL Systems
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Studies selective classification (reject option) for text-to-SQL using entropy-based confidence, Gaussian Mixture/logistic/threshold classifiers, and calibration (MinMax, Platt, isotonic) across distribution-shift splits of SPIDER-based PAUQ and EHRSQL.
Problem
Text-to-SQL models generate incorrect queries under distribution shift (cross-database, compositional, unanswerable questions) without reliable confidence signals, so users cannot tell when to trust outputs.
Approach
Applies selective prediction to text-to-SQL by using maximum token-entropy as the sequence uncertainty and fitting an external selective classifier (Gaussian Mixture, logistic regression, threshold) to flag execution errors, then calibrates scores with Platt and isotonic regression and analyzes coverage-risk trade-offs.
Key results
- Gaussian Mixture selective classifier has the best recall/FDR trade-off (Recall 0.798, FDR 0.364, Result EX 0.251) versus logistic regression (0.873 recall, FDR 0.469) and threshold.
- Isotonic regression gives the lowest Brier scores across models, e.g., T5-3B 0.106 vs MinMax 0.17 and T5-large 0.117 vs 0.2.
- Gaussian Mixture reduces system risk by about 80% on average across splits, but with high false discovery rate under distribution shift; FDR as low as 10% on EHRSQL for some models.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Gaussian Mixture instead of Logistic Regression for selective error detection | Selective text-to-SQL classifier · All splits and models | FDR: 0.469 -> 0.364 | Helped |
| Use Gaussian Mixture instead of Threshold for selective error detection | Selective text-to-SQL classifier · All splits and models | Result EX: 0.143 -> 0.251 | Helped |
| Add Isotonic calibration instead of MinMax normalization | T5-large uncertainty estimates · All splits | Brier score: 0.2 -> 0.117 | Helped |
| Add Isotonic calibration instead of Platt calibration | T5-large uncertainty estimates · All splits | Brier score: 0.121 -> 0.117 | Helped |
| Add Isotonic calibration instead of MinMax normalization | T5-3B uncertainty estimates · All splits | Brier score: 0.17 -> 0.106 | Helped |
| Add Isotonic calibration instead of Platt calibration | T5-3B uncertainty estimates · All splits | Brier score: 0.108 -> 0.106 | Helped |
| Add Isotonic calibration instead of MinMax normalization | Llama3-8B LoRA uncertainty estimates · All splits | Brier score: 0.22 -> 0.199 | Helped |
| Add Isotonic calibration instead of Platt calibration | Llama3-8B LoRA uncertainty estimates · All splits | Brier score: 0.216 -> 0.199 | Helped |
| Add Isotonic calibration instead of MinMax normalization | Llama3-8B SFT uncertainty estimates · All splits | Brier score: 0.21 -> 0.175 | Helped |
| Add Isotonic calibration instead of Platt calibration | Llama3-8B SFT uncertainty estimates · All splits | Brier score: 0.19 -> 0.175 | Helped |
| Add Isotonic calibration instead of MinMax normalization | DAIL-SQL uncertainty estimates · All splits | Brier score: 0.239 -> 0.152 | Helped |
| Add Isotonic calibration instead of Platt calibration | DAIL-SQL uncertainty estimates · All splits | Brier score: 0.16 -> 0.152 | Helped |
Surprising findings
- Selective classifiers more often flag queries to irrelevant (unanswerable) questions than incorrect queries to relevant questions.
- Original entropy-based confidence is not calibrated under distribution shift, and calibration quality does not relate to query complexity.
Limitations
- Analysis limited to SPIDER-based PAUQ and EHRSQL, limiting generalizability.
- Selective classifiers under full distribution shift have high FDR and low coverage.
- Not proposing a new model; focuses on analysis under distribution shifts.
Experimental setup
Datasets: SPIDER PAUQ EHRSQL
Benchmarks: PAUQ XSP Template SSP TSL SSP EHRSQL
Metrics: F-beta score recall false discovery rate AUC Brier score coverage-risk
Baselines: Logistic Regression selective classifier Threshold selective classifier MinMax normalization Platt calibration
Builds on: DAIL-SQL LoRA PAUQ EHRSQL Platt scaling
Closest papers by meaning
- SQLens: An End-to-End Framework for Error Detection and Correction in Text-to-SQL (NeurIPS 2025 · similarity 0.89)
- Alpha-SQL: Zero-Shot Text-to-SQL using Monte Carlo Tree Search (ICML 2025 · similarity 0.86)
- ROUTE: Robust Multitask Tuning and Collaboration for Text-to-SQL (ICLR 2025 · similarity 0.85)
- CogSQL: A Cognitive Framework for Enhancing Large Language Models in Text-to-SQL Translation (AAAI 2025 · similarity 0.85)
- CHASE-SQL: Multi-Path Reasoning and Preference Optimized Candidate Selection in Text-to-SQL (ICLR 2025 · similarity 0.85)
- SQL-R1: Training Natural Language to SQL Reasoning Model By Reinforcement Learning (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.