Research map / Large language models / Agents, code and math
From Passive to Active Reasoning: Can Large Language Models Ask the Right Questions under Incomplete Information?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces AR-Bench, a benchmark of 6,040 puzzles (detective cases, situation puzzles, guessing numbers) that tests whether LLMs can actively ask questions to acquire missing information, and finds current LLMs and common enhancement methods perform poorly.
Problem
LLM evaluations mostly measure passive reasoning with complete information, while active reasoning (acquiring missing information through multi-round interaction) has received little systematic benchmarking.
Approach
A benchmark where a player LLM starts with partial clues and must interrogate NPCs (LLM- or rule-based answerers/judges) over 25 rounds, scored on both final outcome and stepwise key-question coverage, spanning commonsense, lateral-logic, and symbolic tasks.
Key results
- GPT-4o reaches only 35% exact match on GN, and models gain 7.7% process score over rounds 5-10 but only 2.5% over rounds 20-25.
- Llama-3.1-405B achieves 96% accuracy as an LLM judge on 200 annotated SP questions.
- SFT scores 0 on GN and DPO underperforms zero-shot on SP and GN, showing post-training gives little benefit.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use tree-of-thought search instead of zero-shot (ablation) | Llama-3.1-70B · AR-Bench SP | F1 score | Hurt |
| Use supervised fine-tuning (ablation) | Llama-3.1-8B · AR-Bench GN | exact match rate | Hurt |
| Use direct preference optimization instead of zero-shot (ablation) | Llama-3.1-8B · AR-Bench SP and GN | outcome score | Hurt |
| Add human-crafted instructions to prompting (ablation) | LLMs · AR-Bench DC | outcome score | Hurt |
| Add human-crafted instructions to prompting (ablation) | LLMs · AR-Bench SP and GN | outcome score | No clear effect |
| Use Proactive CoT (ablation) | Llama-3.1-8B · AR-Bench SP | outcome score | Helped |
| Use uncertainty-of-thought instead of zero-shot (ablation) | Llama-3.1-8B · AR-Bench | outcome score | Hurt |
| Increase interaction rounds from 25 to 100 (ablation) | Llama-3.1-70B · AR-Bench GN | final score: $8\%$ to $18\% | Helped |
| Increase interaction rounds from 25 to 100 (ablation) | Llama-3.1-70B · AR-Bench DC and SP | final score | No clear effect |
| Continue question asking beyond the first 50 rounds (ablation) | Llama-3.1-70B · AR-Bench process score | process score increase: $45.8\%$ increase in first 50 rounds; $6.7\%$ in the subsequent 50 rounds | Mixed |
| Use a reliable verification function in tree-of-thought search (ablation) | ToT · AR-Bench GN | performance | Helped |
| Use heuristic verifiers in tree-of-thought search (ablation) | ToT · AR-Bench DC and SP | performance | No clear effect |
Surprising findings
- Human-crafted instructions and few-shot examples do not improve active reasoning and can even hurt DC performance.
- Scaling interaction rounds from 25 to 100 yields only limited gains (e.g., GN from 8% to 18%).
Limitations
- Puzzles are synthetic and simplified, with NPC feedback that is not deceptive and may not reflect real-world interactions.
- LLM-based judges (Llama-3.1-405B) are used for SP and DC, so evaluation depends on judge reliability; GN uses an oracle function.
Experimental setup
Datasets: AR-Bench DC training split AR-Bench SP training split AR-Bench GN training split TurtleBenchmark
Benchmarks: AR-Bench DC AR-Bench SP AR-Bench GN TurtleBenchmark
Metrics: accuracy F1 score (character level) exact match rate process score
Baselines: zero-shot few-shot few-shot with instruction tree-of-thought SFT DPO Proactive CoT UoT
Builds on: tree-of-thought direct preference optimization uncertainty-of-thought proactive chain-of-thought
Closest papers by meaning
- AbstentionBench: Reasoning LLMs Fail on Unanswerable Questions (NeurIPS 2025 · similarity 0.90)
- Training Language Models to Reason Efficiently (NeurIPS 2025 · similarity 0.89)
- MDK12-Bench: A Multi-Discipline Benchmark for Evaluating Reasoning in Multimodal Large Language Models (AAAI 2026 · similarity 0.89)
- Exploring the Limits of Sub-Billion Language Model Reasoners with Open Training Recipes (ICLR 2026 · similarity 0.89)
- Meaningful Learning: Enhancing Abstract Reasoning in Large Language Models via Generic Fact Guidance (NeurIPS 2024 · similarity 0.89)
- Eliciting Reasoning in Language Models with Cognitive Tools (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.