Research map / Large language models / Reasoning and chain of thought

LogicCat: A Chain-of-Thought Text-to-SQL Benchmark for Complex Reasoning

AAAI 2026 · Large language models › Reasoning and chain of thought · approach: Search and self-correction in reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces LogicCat, a 4,038-question Text-to-SQL benchmark with chain-of-thought annotations covering physics, arithmetic, commonsense, and hypothetical reasoning, and evaluates LLMs and specialized Text-to-SQL methods on it (best EX 33.20%).

Problem

Existing Text-to-SQL benchmarks do not test multi-step mathematical, physical, commonsense, and hypothetical reasoning that real-world database queries require.

Approach

A benchmark of question-SQL pairs over 45 databases, each annotated with chain-of-thought reasoning steps and tagged by reasoning type (physical, mathematical, commonsense, hypothetical), with difficulty binned by SQL token length and arithmetic operator count.

Key results

• Best overall execution accuracy on LogicCat is 33.20% (Chess-SQL+GPT-4o), with the strongest general LLM at about 30.06% (GPT-4.1).
• Execution accuracy drops from easy to hard questions; the best hard-level EX is 14.96% (CHASE-SQL+Gemini-2.5).
• Removing chain-of-thought lowers EX across all tested models, e.g. Gemini-2.5-Pro from 29.26% to 10.17%.

Surprising findings

• Open-source coder models such as Qwen3-Coder-480B reach strong results with CoT, roughly matching top general LLMs.
• Commonsense questions are the easiest category, while physical and mathematical questions are harder despite the models' pretraining knowledge.

Limitations

• Error analysis is limited to Chess-SQL on 500 sampled examples.
• Reported statistics are inconsistent across sections (e.g., 12,114 vs 12,144 CoT annotations; 35.73% vs 35.0% physical share).

Experimental setup

Datasets: Spider BIRD Archer

Benchmarks: LogicCat

Metrics: execution accuracy valid execution syntax

Baselines: Gemini-2.5-Pro o4-mini o3-mini GPT-4.1 GPT-4o Claude-4.0-Sonnet Claude-3.7-Sonnet Qwen3-235B-A22B Qwen2.5-72B-Instruct DeepSeek-R1 DeepSeek-V3 Kimi-Dev-72B

Builds on: Spider BIRD Archer CHASE-SQL CHESS

Closest papers by meaning

• Alpha-SQL: Zero-Shot Text-to-SQL using Monte Carlo Tree Search (ICML 2025 · similarity 0.88)
• ZebraLogic: On the Scaling Limits of LLMs for Logical Reasoning (ICML 2025 · similarity 0.88)
• CogSQL: A Cognitive Framework for Enhancing Large Language Models in Text-to-SQL Translation (AAAI 2025 · similarity 0.88)
• Chain-of-Table: Evolving Tables in the Reasoning Chain for Table Understanding (ICLR 2024 · similarity 0.88)
• ORACLE: Optimizing Reasoning Abilities of Large Language Models via Constraint-Led Synthetic Data Elicitation (AAAI 2026 · similarity 0.87)
• TableBench: A Comprehensive and Complex Benchmark for Table Question Answering (AAAI 2025 · similarity 0.87)