Research map / Large language models / Agents, code and math
QuestBench: Can LLMs ask the right question to acquire information in reasoning tasks?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces QuestBench, a multiple-choice benchmark formalizing underspecified reasoning problems as 1-sufficient CSPs, testing whether LLMs choose the single right clarifying question across logic, planning, and math.
Problem
LLMs are evaluated on reasoning tasks assuming complete information, but real user queries are often underspecified and require asking for missing information; there is no objective benchmark for this information-gathering ability.
Approach
Formalize underspecification as a constraint satisfaction problem and restrict to 1-sufficient CSPs, where exactly one unknown variable's value suffices to determine the target, so question asking becomes a multiple-choice task with objectively correct answers across Logic-Q, Planning-Q, GSM-Q and GSME-Q.
Key results
- Frontier models exceed 80% accuracy on GSM-Q and GSME-Q (e.g., GPT-4o 96.69% ZS on GSME-Q) but stay under 50% on Logic-Q and Planning-Q (o1-preview 49.91% Logic-Q, 47.32% max Planning-Q for Gemini Flash Thinking).
- Accuracy is negatively correlated with backward search depth, number of variables, and expected brute-force guesses in Logic-Q, but largely uncorrelated with these in Planning-Q.
- Restricting to examples the model solves in well-specified form raises Planning-Q accuracy by up to 23.1% for Gemma 2 27B IT, yet Logic-Q accuracy barely changes, showing reasoning ability alone does not ensure good question asking.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use chain-of-thought prompting instead of zero-shot prompting (ablation) | LLMs evaluated on QuestBench · Logic-Q and Planning-Q | accuracy | Mixed |
| Use four-shot prompting instead of zero-shot prompting (ablation) | LLMs evaluated on QuestBench · Logic-Q and Planning-Q | accuracy | Mixed |
| Present math problems as equations rather than verbalized problems | LLMs evaluated on QuestBench · GSME-Q vs GSM-Q | accuracy | Helped |
| Filter question-asking evaluation to examples whose well-specified version the model can answer (ablation) | LLMs evaluated on Logic-Q · Logic-Q | accuracy | No clear effect |
| Filter question-asking evaluation to examples whose well-specified version the model can answer (ablation) | LLMs evaluated on Planning-Q · Planning-Q | accuracy: 6.5% | Helped |
| Filter question-asking evaluation to examples whose well-specified version the model can answer (ablation) | GPT-4o (ZS) · Planning-Q | accuracy: 8.9% | Helped |
| Filter question-asking evaluation to examples whose well-specified version the model can answer (ablation) | Gemma 2 2B IT (ZS) · GSME-Q | accuracy: 39.0% | Helped |
| Filter question-asking evaluation to examples whose well-specified version the model can answer (ablation) | Gemma 2 9B IT (4S) · GSME-Q | accuracy: -10.3% | Hurt |
| Filter question-asking evaluation to examples whose well-specified version the model can answer (ablation) | Gemini 1.5 Pro (ZS) · GSM-Q | accuracy: 6.2% | Helped |
| Filter question-asking evaluation to examples whose well-specified version the model can answer (ablation) | Gemini 1.5 Flash (ZS) · GSM-Q | accuracy: -5.2% | Hurt |
| Increase problem complexity (search depth, variable count, constraints, or expected brute-force guesses) | LLMs evaluated on Logic-Q · Logic-Q | accuracy | Hurt |
| Increase expected number of brute-force guesses | LLMs evaluated on Planning-Q · Planning-Q | accuracy | Hurt |
Surprising findings
- Strong reasoning on well-specified Logic-Q problems does not translate into better question asking on the corresponding underspecified problems.
- Models rarely predict 'not sure' on Planning-Q (e.g., 1.4% for Gemini Flash Thinking vs 41.8% ground truth), performing at most ~5% above a random baseline at detecting underspecification.
Limitations
- Restricted to 1-sufficient CSPs needing exactly one clarifying question, which may not reflect multi-turn real-world dialogue.
- Multiple-choice format rather than open-ended question generation.
- GSM-Q/GSME-Q evaluated on subsets (288 and 151 tasks) due to evaluation budget.
Experimental setup
Datasets: SimpleLogic PyperPlan Blocks World GSM-Plus GSM8K
Benchmarks: QuestBench (Logic-Q, Planning-Q, GSM-Q, GSME-Q)
Metrics: accuracy Spearman rank correlation F1 for underspecification detection
Baselines: breadth-first search (depth 3, 5, 10) Gemini 1.5 Flash Gemini 1.5 Pro Claude 3.5 Sonnet GPT-4o
Builds on: SimpleLogic PyperPlan GSM-Plus GSM8K PDDL
Closest papers by meaning
- AbstentionBench: Reasoning LLMs Fail on Unanswerable Questions (NeurIPS 2025 · similarity 0.91)
- DOTS: Learning to Reason Dynamically in LLMs via Optimal Reasoning Trajectories Search (ICLR 2025 · similarity 0.88)
- GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models (ICLR 2025 · similarity 0.88)
- Explore What LLM Does Not Know in Complex Question Answering (AAAI 2025 · similarity 0.88)
- Evaluating LLM Reasoning in the Operations Research Domain with ORQA (AAAI 2025 · similarity 0.88)
- From Passive to Active Reasoning: Can Large Language Models Ask the Right Questions under Incomplete Information? (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.