Research map / Large language models / Agents, code and math

CHASE-SQL: Multi-Path Reasoning and Preference Optimized Candidate Selection in Text-to-SQL

ICLR 2025 · Large language models › Agents, code and math · approach: LLM-guided program synthesis · 7 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

CHASE-SQL generates diverse SQL candidates via three LLM generators (divide-and-conquer CoT, query-plan CoT, online synthetic examples) and selects the best with a fine-tuned pairwise selection agent, reaching SOTA on BIRD.

Problem

Text-to-SQL accuracy with LLMs is limited by a single prompt's coverage and by selection methods like self-consistency, which often pick wrong answers even though correct candidates exist (upper bound ~14-20% higher).

Approach

Combine three diverse candidate generators (divide-and-conquer CoT, execution-plan-based CoT, and instance-aware online synthetic few-shot examples) with a pairwise binary selection agent fine-tuned to compare candidates and score them via a round-robin tournament, instead of majority voting.

Key results

• CHASE-SQL with Gemini 1.5 Pro reaches 73.01% execution accuracy on BIRD dev and 73.0% on BIRD test, top of the leaderboard at submission time.
• On Spider test, CHASE-SQL without Spider training achieves 87.6% execution accuracy, second among methods specifically trained or prompt-optimized for Spider.
• The trained selection agent beats self-consistency by 4.17 points in the ablation (73.01 vs 68.84) and the upper bound of combined candidates is 82.79%.

Measured effects reported in the paper

Surprising findings

• Off-the-shelf Gemini-1.5-pro reached only 58.01% binary pairwise selection accuracy without fine-tuning, motivating the trained selector.
• Most-consistent answers are not always correct: self-consistency leaves an upper bound roughly 14-20 points higher on BIRD dev.

Limitations

• Relies on proprietary closed models (Gemini, Claude) with no released code or weights.
• Selection errors remain a major failure mode: correct candidates exist but are not chosen in about 10% of cases.
• Ground-truth SQL used as hint when no correct candidate exists for training data, and BIRD gold queries can be wrong.

Experimental setup

Datasets: BIRD Spider

Benchmarks: BIRD Spider

Metrics: execution accuracy binary selection accuracy

Baselines: Distillery + GPT-4o CHESS MCS-SQL + GPT-4 SuperSQL DAIL-SQL + GPT-4 DIN-SQL + GPT-4 Self-consistency Ranker agent Claude-3.5-sonnet Gemini-1.5-pro

Builds on: chain-of-thought prompting self-consistency self-reflection MCS-SQL

Closest papers by meaning

• Alpha-SQL: Zero-Shot Text-to-SQL using Monte Carlo Tree Search (ICML 2025 · similarity 0.89)
• MARS-SQL: A Multi-Agent Reinforcement Learning Framework For Text-To-SQL (ICML 2026 · similarity 0.89)
• ROUTE: Robust Multitask Tuning and Collaboration for Text-to-SQL (ICLR 2025 · similarity 0.88)
• Structure-Guided Large Language Models for Text-to-SQL Generation (ICML 2025 · similarity 0.87)
• CogSQL: A Cognitive Framework for Enhancing Large Language Models in Text-to-SQL Translation (AAAI 2025 · similarity 0.87)
• LogicCat: A Chain-of-Thought Text-to-SQL Benchmark for Complex Reasoning (AAAI 2026 · similarity 0.86)