Research map / Multi-agent systems and games / LLM agents

MedEvoEval: Evaluating Continual Evolution of Doctor Agents through Simulated Clinical Episodes

NeurIPS 2026 · Multi-agent systems and games › LLM agents · approach: Agentic benchmarks and orchestration · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces MedEvoEval, an executable longitudinal evaluation framework that runs action-gated simulated outpatient episodes for doctor agents, tracking process cost, MDT consultation effects, memory maturation, held-out transfer, and backward retention, with a released 700-case Chinese MedQA-derived artifact.

Problem

Existing medical QA benchmarks give full vignettes upfront and score only final answers, while interactive benchmarks mostly evaluate single encounters; they do not measure how evidence acquisition, resource use, and experience accumulation across episodes interact.

Approach

Converts each source case into patient, examination, and manager views, reveals evidence only through gated ASK/REQUEST_TEST actions, logs structured event traces, and runs longitudinal memory write-back with held-out transfer (H0 vs H3) and BWT-based retention analysis.

Key results

• GLM-5 reaches trace S 0.622 and strict accuracy 0.375 on 80 episodes, versus 0.304 and 0.100 for Qwen-Turbo.
• Mature memory (E3) raises stream total score from 0.246 (E0) to 0.430 and cuts turns from 6.134 to 5.380.
• On 100 unseen held-out cases, mature memory improves total score from 0.338 to 0.413 and reduces turns by 0.99; BWT for score is -0.0087 indicating mild backward degradation.

Surprising findings

• Qwen-Turbo improves under full-vignette static input (S 0.585) relative to its dynamic episode score (0.304), while Kimi-K2.5 has higher static composite but lower static strict accuracy.
• Under MDT consultation, examination use drops sharply while strict accuracy and total score remain close across conditions.

Limitations

• Episodes are simulated with structured patient views and pre-specified examination results; manager scores depend on rubrics and automatic judging.
• Memory results may overestimate in-distribution gains despite held-out transfer; the update event is a retention probe, not a full robustness benchmark.
• Baselines cover only four API-accessible models, with no open-source model families or scale sweeps.

Experimental setup

Datasets: MedQA (Chinese, source cases converted to Chinese MedQA-700)

Benchmarks: Chinese MedEvoEval-MedQA-700

Metrics: outcome score S utility U strict diagnostic accuracy pass rate turns tests invalid-test rate BWT ET

Baselines: Qwen-Turbo Qwen3.5-122B-A10B GLM-5 Kimi-K2.5

Builds on: MedAgentSim MedQA

Closest papers by meaning

• Experience Makes Skillful: Enabling Generalizable Medical Agent Reasoning via Self-Evolving Skill Memory (NeurIPS 2026 · similarity 0.88)
• Doctor-R1: Mastering Clinical Inquiry with Experiential Agentic Reinforcement Learning (ICLR 2026 · similarity 0.88)
• Simulating Viva Voce Examinations to Evaluate Clinical Reasoning in Large Language Models (NeurIPS 2025 · similarity 0.87)
• MedExAgent: Training LLM Agents to Ask, Examine, and Diagnose in Noisy Clinical Environments (NeurIPS 2026 · similarity 0.87)
• PhysicianBench: Evaluating LLM Agents in Real-World EHR Environments (NeurIPS 2026 · similarity 0.87)
• MedAgentBoard: Benchmarking Multi-Agent Collaboration with Conventional Methods for Diverse Medical Tasks (NeurIPS 2025 · similarity 0.86)