Research map / Large language models / Agents, code and math
Active Task Disambiguation with LLMs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes active task disambiguation, selecting clarifying questions by estimating expected information gain over self-sampled candidate solutions, and shows it beats implicit question generation on a 20-questions game and code generation.
Problem
LLM agents often face ambiguously specified tasks, but generating informative clarifying questions requires meta-cognitive reasoning about their own solution distribution, which LLMs appear to lack.
Approach
Frame task disambiguation as Bayesian Experimental Design: sample candidate solutions, sample candidate questions, simulate answers to each question over the sampled solutions, and pick the question with maximum estimated expected information gain (shifting reasoning from question space to solution space).
Key results
- On the 20-questions game with GPT-3.5-turbo, EIG-uniform reaches an average rank of 6.2 at iteration 10 versus 4.9 for implicit (out of 10).
- On HumanEval with GPT-3.5-turbo, open EIG questions raise accuracy after 4 queries from 75.9% (base open) to 85.0%.
- On APPS with GPT-4o-mini, open EIG questions raise accuracy after 4 queries from 75.1% to 87.2%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Select open coding questions by estimated EIG rather than generating them zero-shot | GPT-3.5-turbo code generation · HumanEval, t=4 | accuracy (%): 75.9 -> 85.0 | Helped |
| Select binary coding questions by estimated EIG rather than generating them zero-shot | GPT-3.5-turbo code generation · HumanEval, t=4 | accuracy (%): 70.8 -> 85.6 | Helped |
| Select open coding questions by estimated EIG rather than generating them zero-shot | GPT-4o-mini code generation · APPS, t=4 | accuracy (%): 75.1 -> 87.2 | Helped |
| Select binary coding questions by estimated EIG rather than generating them zero-shot | GPT-4o-mini code generation · APPS, t=4 | accuracy (%): 62.9 -> 75.3 | Helped |
| Select animal questions with uniform-sample EIG rather than implicit reasoning | GPT-3.5-turbo · 20 Questions, t=10 | average rank: 4.9 -> 6.2 | Helped |
| Estimate EIG using solution log-probabilities instead of uniform solution weights | GPT-3.5-turbo · 20 Questions, t=10 | average rank: 6.2 -> 4.1 | Hurt |
| Add a diverse-and-representative instruction when sampling solutions (ablation) | GPT-3.5-turbo · 20 Questions, N=10 | conditional MI: 0.46 -> 0.57 | Helped |
| Add a diverse-and-representative instruction when sampling solutions (ablation) | GPT-3.5-turbo · 20 Questions, N=20 | conditional MI: 0.54 -> 0.58 | Helped |
| Add a diverse-and-representative instruction when sampling solutions (ablation) | GPT-3.5-turbo · 20 Questions, N=30 | conditional MI: 0.58 -> 0.60 | Helped |
| Filter sampled animals against current requirements (ablation) | EIG-uniform, GPT-3.5-turbo · 20 Questions | final ranking | Helped |
| Use open questions instead of binary questions | EIG-based code elicitation · HumanEval and APPS | accuracy (%) | Mixed |
Surprising findings
- EIG estimated from raw log-probabilities of sampled solutions (EIG-logprobs) underperforms uniform-weighted EIG, suggesting LLM generative distributions are biased away from the ground-truth solution set.
- The gap between EIG and implicit strategies is much smaller for the more capable GPT-4o-mini on the 20-questions game.
Limitations
- Requires many extra LLM calls per question (sampling solutions, questions, and simulated answers), increasing cost.
- Assumes uniform approximation of the unknown solution distribution and error-free answers, which may be violated when answering from LLM self-simulation.
- Evaluated on a small set of tasks (15 animals, 48 code tasks) with only a constant cost assumption.
Experimental setup
Datasets: HumanEval APPS
Benchmarks: HumanEval APPS 20 questions animal game
Metrics: average rank of ground-truth animal code accuracy on hidden tests number of compatible animals
Baselines: implicit question generation implicit-ToT EIG-logprobs zero-shot binary test case questions
Builds on: tree of thoughts active learning Bayesian experimental design
Closest papers by meaning
- Distilling LLM Agent into Small Models with Retrieval and Code Tools (NeurIPS 2025 · similarity 0.88)
- From Passive to Active Reasoning: Can Large Language Models Ask the Right Questions under Incomplete Information? (ICML 2025 · similarity 0.88)
- Language Agent Tree Search Unifies Reasoning, Acting, and Planning in Language Models (ICML 2024 · similarity 0.87)
- Chain of Agents: Large Language Models Collaborating on Long-Context Tasks (NeurIPS 2024 · similarity 0.87)
- DOTS: Learning to Reason Dynamically in LLMs via Optimal Reasoning Trajectories Search (ICLR 2025 · similarity 0.87)
- Probing the Trajectories of Reasoning Traces in Large Language Models (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.