Research map / Large language models / Language, safety and interpretability
AMBROSIA: A Benchmark for Parsing Ambiguous Questions into Database Queries
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces AMBROSIA, a text-to-SQL benchmark of ambiguous questions (scope, attachment, vagueness) over LLM-generated multi-table databases, and shows that LLMs largely fail to produce all interpretations.
Problem
Semantic parsers must map ambiguous user questions to all valid SQL interpretations, but existing text-to-SQL benchmarks mostly lack ambiguous questions that remain ambiguous given the database context.
Approach
Controlled, LLM-driven generation of realistic multi-table databases from key concepts/relations templates designed so that specific ambiguity types persist, paired with human-written ambiguous questions, interpretations, and SQL, and evaluated with recall/precision/AllFound metrics.
Key results
- Llama3-70B with Prompt reaches 30.7% recall on ambiguous questions versus 64.5% on unambiguous ones in zero-shot.
- Best AllFound on ambiguous questions is only 1.9% zero-shot (Llama3-70B), rising to 4.5% for GPT-4o one-shot.
- Llama3-70B predicts the distributive scope reading in 83.5% of cases and high attachment in 98.2%, showing strong interpretation bias.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add one in-context example | Llama3-70B Prompt · AMBROSIA ambiguous questions | recall: 30.7 -> 32.4 | Helped |
| Add one in-context example | Llama3-70B Prompt · AMBROSIA ambiguous questions | AllFound: 1.9 -> 3.7 | Helped |
| Add one in-context example | GPT-4o Prompt · AMBROSIA ambiguous questions | recall: 27.1 -> 31.3 | Helped |
| Add one in-context example | GPT-4o Prompt · AMBROSIA ambiguous questions | AllFound: 0.4 -> 4.5 | Helped |
| Add one in-context example | Llama3-70B Prompt · AMBROSIA ambiguous questions | precision: 42.7 -> 51.7 | No clear effect |
| Include definitions and examples for all three ambiguity types instead of randomly selected examples | Llama3-70B Prompt · AMBROSIA ambiguous questions | recall: 30.7 -> 35.0 | Helped |
| Use Beam top-5 predictions instead of Prompt instructions | Llama3-70B · AMBROSIA ambiguous questions | recall: 30.7 -> 28.0 | Hurt |
| Use Beam top-5 predictions instead of Prompt instructions | CodeLlama-70B · AMBROSIA ambiguous questions | recall: 17.9 -> 25.4 | Helped |
| Use Prompt rather than Beam predictions | Llama3-70B · AMBROSIA ambiguous questions | AllFound: 1.4 -> 1.9 | Helped |
| Use Prompt rather than Beam predictions | Llama3-70B · AMBROSIA ambiguous questions | recall: Scope: 41.6 -> 41.5; Attachment: 10.3 -> 12.7; Vague: 25.8 -> 35.6 | Mixed |
Surprising findings
- Even the strongest models rarely produce all interpretations for ambiguous questions, often giving one correct SQL and missing others.
- Attachment ambiguity is hardest, with only 12.7% ambiguous recall for Llama3-70B, even though unambiguous recall is also low at 24.0%.
Limitations
- Databases are simple and clean, unlike real-world databases with abbreviations or incomplete content.
- Experiments provide full database content in the prompt, which is not scalable or safe for real deployments, so results are an upper bound.
- The dataset does not exhaustively cover all ambiguity types, e.g., lexical ambiguity is excluded.
Experimental setup
Datasets: AMBROSIA
Benchmarks: AMBROSIA
Metrics: recall precision AllFound execution accuracy
Baselines: OpenChat-7B Llama3-8B CodeLlama-70B Llama3-70B GPT-3.5 Turbo GPT-4o
Builds on: Spider BIRD-Bench AmbiQT NoisySP
Closest papers by meaning
- Zero and Few-shot Semantic Parsing with Ambiguous Inputs (ICLR 2024 · similarity 0.86)
- CogSQL: A Cognitive Framework for Enhancing Large Language Models in Text-to-SQL Translation (AAAI 2025 · similarity 0.85)
- SQLens: An End-to-End Framework for Error Detection and Correction in Text-to-SQL (NeurIPS 2025 · similarity 0.84)
- SQLFixAgent: Towards Semantic-Accurate Text-to-SQL Parsing via Consistency-Enhanced Multi-Agent Collaboration (AAAI 2025 · similarity 0.84)
- Structure-Guided Large Language Models for Text-to-SQL Generation (ICML 2025 · similarity 0.84)
- Filling Memory Gaps: Enhancing Continual Semantic Parsing via SQL Syntax Variance-Guided LLMs Without Real Data Replay (AAAI 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.