Research map / Multi-agent systems and games / LLM agents
From Conversation to Query Execution: Benchmarking User and Tool Interactions for EHR Database Agents
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces EHR-ChatQA, an interactive EHR database QA benchmark with LLM-simulated users and tools, evaluating agents on query ambiguity and value mismatch across IncreQA and AdaptQA flows with Pass@k vs Pass^k consistency metrics.
Problem
Existing text-to-SQL and EHR QA benchmarks use single well-formed questions and do not capture query ambiguity and value mismatch in interactive clinical database access.
Approach
A simulated-user benchmark over renamed MIMIC-IV and eICU demo databases where agents must clarify vague questions, use schema/value/web tools, and write SQL, with a simulation validator filtering invalid user behavior and reporting Pass@5 versus Pass^5 consistency.
Key results
- o4-mini reaches Pass@5 of 96.5% on IncreQA but Pass^5 of only 57.7% (Gap-5 38.8%).
- On AdaptQA, o4-mini achieves Pass@5 66.2% with Pass^5 7.5% (Gap-5 58.7%).
- Gemini-2.5-Flash achieves SR-5 of 73.3% on IncreQA and 30.0% on AdaptQA.
Surprising findings
- Best agents achieve high Pass@5 but very low Pass^5 on AdaptQA, with gaps up to about 60%.
- Renaming database schemas was needed because SOTA LLMs memorized original MIMIC-IV and eICU schemas.
Limitations
- Benchmark uses demo versions of MIMIC-IV and eICU with only 366 tasks.
- LLM-based user simulator is not fully instruction-reliable; validator has 10% false rejection and 9% false acceptance.
- Open-source models served locally and only two evaluated; agent architecture not optimized.
Experimental setup
Datasets: MIMIC-IV demo eICU demo EHRSQL
Benchmarks: EHR-ChatQA
Metrics: SR-5 Pass@5 Pass^5 Gap-5
Baselines: o4-mini GPT-4o GPT-4o-mini Gemini-2.5-Flash Gemini-2.0-Flash Llama 3.3-70B Qwen3-32B
Builds on: Tau-Bench EHRSQL Pass^k
Closest papers by meaning
- Instruction Tuning Large Language Models to Understand Electronic Health Records (NeurIPS 2024 · similarity 0.87)
- MediQ: Question-Asking LLMs and a Benchmark for Reliable Interactive Clinical Reasoning (NeurIPS 2024 · similarity 0.86)
- SM3-Text-to-Query: Synthetic Multi-Model Medical Text-to-Query Benchmark (NeurIPS 2024 · similarity 0.86)
- MedAlign: A Clinician-Generated Dataset for Instruction Following with Electronic Medical Records (AAAI 2024 · similarity 0.85)
- Reinforcement Learning for Tool-Calling Agents in Fast Healthcare Interoperability Resources (FHIR) (ICML 2026 · similarity 0.85)
- PhysicianBench: Evaluating LLM Agents in Real-World EHR Environments (NeurIPS 2026 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.