Research map / Large language models / Retrieval and knowledge
MedBench: A Large-Scale Chinese Benchmark for Evaluating Medical Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces MedBench, a 40,041-item Chinese medical benchmark built from three-stage licensing/training exams and expert-annotated EHR cases, and evaluates general and Chinese medical LLMs on it.
Problem
Existing medical LLM benchmarks rely on open-access, possibly contaminated exam data, are largely English or USMLE/Indian-centric, and miss China's three-stage physician examination pipeline and real-world clinical case evaluation.
Approach
A Chinese medical benchmark combining authentic CNMLE, Resident Standardization Training, and Doctor in-charge Qualification exams with expert-annotated electronic health record case QA, plus an IRT-3PL-based difficulty stratification to reduce evaluation cost and better separate models.
Key results
- GPT-4 scores 64.88% total on CNMLE and 68.45% on Doctor in-Charge Qualification, the top results in Table 1; ChatGPT ~50% on CNMLE.
- Chinese medical LLMs (HuaTuo 22.31%, ChatMed 23.45% on CNMLE) underperform strongly relative to general-domain models.
- Human evaluation by a postgraduate medical scholar rates GPT-4 best across correctness, completeness, fluency, and friendliness on real-world cases.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Chain-of-Thought prompting instead of a vanilla prompt (ablation) | Baichuan-13B · Resident Standardization Training Exam, A1/A2 | accuracy: 37.26 -> 43.85 | Helped |
| Use Chain-of-Thought prompting instead of a vanilla prompt (ablation) | ChatGLM-6B · Resident Standardization Training Exam, A1/A2 | accuracy: 42.69 -> 43.37 | No clear effect |
Surprising findings
- Several general-domain LLMs (e.g., ChatGPT, GPT-4) outperform Chinese medical LLMs despite the latter being trained on medical data.
- Baichuan-13B under vanilla prompting heavily favors an invalid option F; chain-of-thought prompting markedly improves its accuracy (37.26% to 43.85%).
Limitations
- Real-world case evaluation relies on BLEU/ROUGE that the authors acknowledge may not capture quality, plus a single human rater.
- Some comparisons are not significant; several Chinese medical LLMs output invalid answers to multiple-choice questions and were excluded.
- Clinical cases lack patient history and physical exam narratives, limiting diagnostic evaluation.
Experimental setup
Datasets: CNMLE questions Resident Standardization Training Exam questions Doctor in-charge Qualification Exam questions expert-annotated electronic health record cases
Benchmarks: MedBench
Metrics: accuracy BLEU ROUGE-F1 human evaluation (correctness, completeness, fluency, friend
Baselines: ChatGPT GPT-4 ChatGLM Baichuan-13B HuaTuo ChatMed MedLLaMA Baize
Builds on: Item Response Theory MedQA CMExam
Closest papers by meaning
- MedJourney: Benchmark and Evaluation of Large Language Models over Patient Clinical Journey (NeurIPS 2024 · similarity 0.92)
- Medmarks: A Comprehensive Open-Source LLM Benchmark Suite for Medical Tasks (NeurIPS 2026 · similarity 0.90)
- Position: Medical Large Language Model Benchmarks Should Prioritize Construct Validity (ICML 2025 · similarity 0.90)
- MedCalc-Bench: Evaluating Large Language Models for Medical Calculations (NeurIPS 2024 · similarity 0.90)
- Zhongjing: Enhancing the Chinese Medical Capabilities of Large Language Model through Expert Feedback and Real-World Multi-Turn Dialogue (AAAI 2024 · similarity 0.90)
- TCM-Ladder: A Benchmark for Multimodal Question Answering on Traditional Chinese Medicine (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.