Research map / Large language models / Agents, code and math
Simulating Viva Voce Examinations to Evaluate Clinical Reasoning in Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces VivaBench, a multi-turn benchmark of 990 physician-curated PubMed/MedQA case vignettes that simulates viva voce clinical examinations, and evaluates six LLMs on sequential diagnostic reasoning.
Problem
Current medical LLM benchmarks test knowledge recall with single-turn questions where complete information is given upfront, missing the hypothesis-driven, sequential information gathering central to clinical reasoning.
Approach
Operationalizes the viva voce oral exam as an interactive benchmark: structured clinical cases are served through an examiner module that returns history, exam, imaging and lab results only when requested, with provisional and final diagnoses scored against ICD-10 ground truth and confidence-weighted accuracy.
Key results
- Gemini 2.5 Pro achieved the highest final-diagnosis top-1 accuracy of 35% versus 69% top-1 with full information.
- Most models at least doubled accuracy when given full information compared with the interactive final-diagnosis stage.
- Failure analysis attributed 348 cases to inappropriate hypothesis generation and 291 to premature diagnostic closure, versus 90 to inadequate investigations.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Provide complete clinical information upfront instead of requiring interactive information gathering | Evaluated LLMs · VivaBench | top-1 exact diagnostic accuracy: Gemini 2.5 Pro: 0.35 -> 0.69 | Helped |
| Require interactive examination rather than provide complete clinical information | Evaluated LLMs · VivaBench | top-1 exact diagnostic accuracy: Gemini 2.5 Pro: 0.69 -> 0.35 | Hurt |
Surprising findings
- Models showed higher precision than recall in information seeking, and ordered many niche tests such as genetic testing rather than basic routine tests.
- Confidence changes correlated with accuracy gains mainly in magnitude, and maintained diagnoses gained confidence regardless of correctness, suggesting confirmation bias.
Limitations
- Dataset derives from a single source type (case reports) and is modest in scale.
- LLM-based mapper is not fully deterministic, and deterministic mapper does not capture full variability of clinical communication.
- Only a single evaluation run per model was conducted due to computational constraints.
Experimental setup
Datasets: MedQA PubMed case reports MIMIC
Benchmarks: VivaBench
Metrics: top-k accuracy confidence-weighted accuracy (S_conf) precision recall number of interaction actions
Baselines: Gemini 2.5 Pro DeepSeek-R1 o4-mini Llama-4 Maverick Grok 3 Mini Beta Qwen 3 (235b-a22b)
Builds on: AgentClinic AI Hospital MedQA
Closest papers by meaning
- MedExAgent: Training LLM Agents to Ask, Examine, and Diagnose in Noisy Clinical Environments (NeurIPS 2026 · similarity 0.89)
- Large Language Models Are Clinical Reasoners: Reasoning-Aware Diagnosis Framework with Prompt-Generated Rationales (AAAI 2024 · similarity 0.88)
- CARE: Towards Clinical Accountability in Multi-Modal Medical Reasoning with an Evidence-Grounded Agentic Framework (ICLR 2026 · similarity 0.88)
- MTBBench: A Multimodal Sequential Clinical Decision-Making Benchmark in Oncology (NeurIPS 2025 · similarity 0.88)
- MediQ: Question-Asking LLMs and a Benchmark for Reliable Interactive Clinical Reasoning (NeurIPS 2024 · similarity 0.88)
- ER-Reason: A Benchmark Dataset for LLM Clinical Reasoning in the Emergency Room (NeurIPS 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.