Research map / Vision-language and multimodal / CLIP and zero-shot
Zero and Few-shot Semantic Parsing with Ambiguous Inputs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces AmP, a templated dataset and few-shot evaluation framework that tests whether text-to-FOL/Lisp semantic parsers capture the distribution over readings of five ambiguity types.
Problem
Semantic parsing datasets typically assign one gold logical form per utterance, so it is unclear whether LLM-based parsers capture the multiple valid interpretations of ambiguous inputs.
Approach
Generate ambiguous utterances from templates for five linguistic ambiguity types, each paired with two first-order-logic readings, and evaluate in-context parsers in a zero-shot compositional setting and a mixed-prompt setting using three new metrics (ZM_k, FDM, FIM).
Key results
- In zero-shot setting, most models score 0.0 on ZM_5, rarely predicting both readings in their top-5 outputs.
- Few-shot mixed prompts: CodeGen-16B and Vicuna-13B achieve low FDM (e.g., 0.35 and 0.50 on PP) and FIM around 0.06-0.09, indicating prompt distributions are tracked.
- Llama-13B gets FDM 1.00 on all ambiguity types because it never predicts either LF correctly, despite low FIM.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase the number of LF_0 examples in mixed prompts | few-shot parsing models · Scope and inverse scope | LF_0 accuracy | Helped |
| Scale Codegen from 350M to 16B parameters | Codegen · PP | FDM: 0.76 -> 0.35 | Helped |
| Instruction-tune Llama-13B as Vicuna-13B | Llama-13B · Scope | FDM: 1.00 -> 0.17 | Helped |
| Render parses in FOL rather than Lisp | Codegen-2B zero-shot parser · Scope | LF_1 accuracy: 11% -> 18% | Helped |
| Provide mixed prompts with examples of both interpretations instead of zero-shot ingredients only | few-shot parsing models · Scope and inverse scope | alignment with prompt distribution | Helped |
| Use larger Codegen models | Codegen · Most ambiguity types | FDM | Helped |
| Use forced decoding to score the gold interpretations | Llama-13B · All ambiguity types | FIM | Helped |
| Use instruction tuning | Llama-13B zero-shot parser · Zero-shot ambiguity parsing | accuracy | Helped |
| Increase model size to Codegen-16B | Codegen few-shot parser · Four of five ambiguity types | FIM | No clear effect |
| Use mixed prompts to expose ambiguity | few-shot parsing models · Few-shot parsing | FIM | Helped |
Surprising findings
- Models fail to produce both readings zero-shot yet track conflicting prompt distributions closely in few-shot settings.
- Llama-13B has low FIM despite never producing a correct LF under beam search, showing FIM and accuracy diverge.
Limitations
- Fixed English-only set of five ambiguity types and a single chosen logical form representation.
- Models do no online processing and lack conversational context, unlike human interpretation.
- Human judgement task (verbalized interpretations) differs from the model's parsing task.
Experimental setup
Datasets: AmP (templated synthetic ambiguous parses)
Benchmarks: AmP
Metrics: ZM_k FDM FIM exact-match accuracy
Baselines: Llama-13B Vicuna-13B gpt-3.5-turbo CodeGen-350M CodeGen-2B CodeGen-6B CodeGen-16B
Builds on: neo-Davidsonian event semantics BenchCLAMP Vicuna Brier score
Closest papers by meaning
- AMBROSIA: A Benchmark for Parsing Ambiguous Questions into Database Queries (NeurIPS 2024 · similarity 0.86)
- Mapping from Meaning: Addressing the Miscalibration of Prompt-Sensitive Language Models (AAAI 2025 · similarity 0.85)
- Emergent Representations of Program Semantics in Language Models Trained on Programs (ICML 2024 · similarity 0.84)
- Large language models suffer from a curse of ambiguity (NeurIPS 2026 · similarity 0.84)
- Evaluating the Zero-shot Robustness of Instruction-tuned Language Models (ICLR 2024 · similarity 0.84)
- SUGARCREPE++ Dataset: Vision-Language Model Sensitivity to Semantic and Lexical Alterations (NeurIPS 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.