Research map / Large language models / Retrieval and knowledge
MediQ: Question-Asking LLMs and a Benchmark for Reliable Interactive Clinical Reasoning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MediQ, an interactive clinical benchmark where a patient simulator answers follow-up questions from an Expert LLM that must decide when to ask versus answer, using abstention strategies to improve diagnostic accuracy over basic question-asking.
Problem
Static single-turn medical QA benchmarks assume complete information, whereas real clinical interactions involve incomplete initial information requiring proactive follow-up questions; LLMs struggle to ask informative questions and decide when to ask.
Approach
Convert MedQA and Craft-MD into interactive settings with partial initial information, pair a fact-grounded Patient system (Fact-Select) with an Expert system that uses confidence-based abstention (numerical, binary, scale) plus rationale generation and self-consistency to decide when to ask follow-up questions.
Key results
- Basic question-asking degrades accuracy versus non-interactive Initial by about 11.3% relative on average across LLMs (e.g., GPT-3.5 on iMedQA 42.2 vs 45.6).
- Scale abstention with rationale generation and self-consistency factor 3 improves over Basic by 22.3% and over non-interactive Initial by 12.1% for GPT-3.5.
- Fact-Select patient system raises factuality by 0.33 points over Direct and Instruct variants, and won 63.77% of manual pairwise comparisons.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Provide full patient information upfront instead of initial information only | Expert systems · iMedQA and iCraft-MD | accuracy | Helped |
| Allow follow-up questions with Basic prompting instead of answering from initial information | Expert systems · iMedQA and iCraft-MD | accuracy: 11.3% relative drop; 7.43% for GPT-3.5 on iMedQA | Hurt |
| Use abstention strategies with rationale generation and self-consistency | MediQ-Expert · iMedQA and iCraft-MD | accuracy: 22.3% over Basic; 12.1% over non-interactive Initial | Helped |
| Add rationale generation to Numerical, Binary, or Scale abstention (ablation) | MediQ-Expert · iMedQA | accuracy | Helped |
| Add self-consistency without rationale generation (ablation) | MediQ-Expert · iMedQA | accuracy | Hurt |
| Add self-consistency to rationale generation (ablation) | MediQ-Expert · iMedQA | accuracy | Helped |
| Use Fact-Select patient responses instead of Direct or Instruct responses (ablation) | Patient system · MediQ patient response evaluation | factuality: +0.33 points | Helped |
| Use Fact-Select patient responses instead of Direct or Instruct responses (ablation) | Patient system · MediQ patient response evaluation | relevance: +0.04 points | Helped |
| Filter unanswerable question-response pairs (ablation) | Basic Expert conversation · iMedQA | accuracy: 2 percentage points | Helped |
| Remove repeated questions (ablation) | Basic Expert conversation · iMedQA | accuracy: 2 percentage points | Helped |
| Filter unanswerable and repeated questions together (ablation) | Basic Expert conversation · iMedQA | accuracy | No clear effect |
| Reformat conversation as paragraphs (ablation) | Basic Expert conversation · iMedQA | accuracy | Helped |
Surprising findings
- Directly prompting state-of-the-art LLMs to ask questions degrades accuracy compared with answering from the same limited initial information.
- For most samples, no model chooses to ask any follow-up question in the Basic setting.
Limitations
- Patient system relies on a paid closed-source API, with no open-source patient model.
- Evaluation limited to multiple-choice format.
- Few medical datasets contain sufficiently detailed patient records; only MedQA and Craft-MD were used.
Experimental setup
Datasets: MedQA Craft-MD
Benchmarks: iMedQA iCraft-MD
Metrics: accuracy number of follow-up questions factuality score relevance score win rate expected calibration error
Baselines: Basic question-asking Llama-3-8B-Instruct Llama-3-70B-Instruct GPT-3.5 GPT-4
Builds on: MedQA Craft-MD chain-of-thought self-consistency
Closest papers by meaning
- Medmarks: A Comprehensive Open-Source LLM Benchmark Suite for Medical Tasks (NeurIPS 2026 · similarity 0.88)
- MedJourney: Benchmark and Evaluation of Large Language Models over Patient Clinical Journey (NeurIPS 2024 · similarity 0.88)
- Simulating Viva Voce Examinations to Evaluate Clinical Reasoning in Large Language Models (NeurIPS 2025 · similarity 0.88)
- MedExAgent: Training LLM Agents to Ask, Examine, and Diagnose in Noisy Clinical Environments (NeurIPS 2026 · similarity 0.88)
- MedChain: Bridging the Gap Between LLM Agents and Clinical Practice with Interactive Sequence (NeurIPS 2025 · similarity 0.87)
- EHRNote-ChatQA: A Benchmark for Evidence-Grounded Multi-Turn Clinical Question Answering over Longitudinal Discharge Summaries (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.