Research map / Large language models / Language, safety and interpretability
Adaptive Elicitation of Latent Information Using Natural Language
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes an adaptive elicitation framework that meta-trains an LLM on historical question-answer trajectories to simulate future answers and select informative questions via expected information gain (greedy or MCTS), evaluated on Twenty Questions, OpinionQA, and EEDI.
Problem
Eliciting information to reduce uncertainty about a new latent entity (student, survey respondent, hidden object) through natural-language questioning; LLMs and fine-tuning methods do not strategically gather information to reduce their uncertainty about such entities.
Approach
Adopt a predictive view of uncertainty: meta-train an autoregressive LLM on historical question-answer sequences (with permuted order and masked answer loss) so it can simulate future answers, then compute expected information gain over target questions by forward simulation and choose questions greedily or via MCTS.
Key results
- Across OpinionQA, EEDI and Twenty Questions, the EIG-based adaptive method outperforms random selection and ICT baselines in target-question accuracy and perplexity, averaged over 10,000 simulated trials.
- On hard target questions, EIG/MCTS gains over random exceed 10x the all-question gain for EEDI and Twenty Questions (5x for OpinionQA).
- Llama-3.1-8B meta-training reaches 0.7382 accuracy on Twenty Questions after 4 questions, versus 0.5201 with GPT2 and 0.6131 with Llama-3.2-1B.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase the candidate question pool from 10 to 25 (ablation) | Adaptive elicitation on OpinionQA · OpinionQA (4 questions) | accuracy: 0.5933 -> 0.6068 | Helped |
| Vary the number of target questions from 1 to 20 (ablation) | Adaptive elicitation on EEDI · EEDI (4 questions) | accuracy: 0.6759 -> 0.6832 | No clear effect |
| Use a larger base model for meta-training (ablation) | Meta-trained model on Twenty Questions · Twenty Questions (4 questions) | accuracy: 0.5201 -> 0.6131 -> 0.7382 | Helped |
| Use EIG-based planning instead of random selection (ablation) | Base LLM · Twenty Questions (hard targets, 3 selected questions) | accuracy | Hurt |
| Use EIG-based planning instead of random selection (ablation) | ICT model · Twenty Questions (all, medium, and hard targets; 3 selected questions) | accuracy | No clear effect |
| Use EIG-based planning instead of random selection (ablation) | Meta-trained model · Twenty Questions (all, medium, and hard targets; 3 selected questions) | accuracy | Helped |
| Use EIG or MCTS rather than random question selection | Adaptive elicitation · EEDI and Twenty Questions (hard targets) | relative accuracy gain: more than 10x higher for hard questions than for all questions | Helped |
| Use EIG or MCTS rather than random question selection | Adaptive elicitation · OpinionQA (hard targets) | relative accuracy gain: 5x higher | Helped |
| Use MCTS instead of greedy EIG selection | Meta-trained adaptive elicitation · OpinionQA, EEDI, and Twenty Questions | relative accuracy gain | Helped |
Surprising findings
- Planning with the base LLM (no meta-training) reduces target accuracy by almost 15% on hard questions relative to random selection.
- ICT model performance is largely unchanged by planning, indicating the meta-training procedure is needed for planning to help.
Limitations
- Random permutation of question-answer pairs assumes answer independence from question ordering, which the authors acknowledge may be unrealistic.
- Submodularity of the meta-learned model's entropy is assumed for the greedy guarantee and is not guaranteed in practice.
- Twenty Questions answers are generated by Claude-3.5-Sonnet rather than real human responses.
Experimental setup
Datasets: Twenty Questions (new) OpinionQA EEDI
Benchmarks: Twenty Questions OpinionQA EEDI
Metrics: accuracy perplexity expected calibration error Brier score
Baselines: Base LLM (Llama-3.1-8B, random selection) In-Context Tuning (ICT) with embedding-similarity question selection
Builds on: In-Context Tuning Expected Information Gain Monte Carlo Tree Search
Closest papers by meaning
- To Believe or Not to Believe Your LLM: Iterative Prompting for Estimating Epistemic Uncertainty (NeurIPS 2024 · similarity 0.87)
- Evaluating language models as risk scores (NeurIPS 2024 · similarity 0.87)
- Improving Uncertainty Estimation through Semantically Diverse Language Generation (ICLR 2025 · similarity 0.87)
- Distinguishing the Knowable from the Unknowable with Language Models (ICML 2024 · similarity 0.87)
- Doing Experiments and Revising Rules with Natural Language and Probabilistic Reasoning (NeurIPS 2024 · similarity 0.87)
- Efficient semantic uncertainty quantification in language models via diversity-steered sampling (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.