Research map / Large language models / AI and society
Position: Medical Large Language Model Benchmarks Should Prioritize Construct Validity
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Position paper arguing medical LLM benchmarks should be empirically validated for construct validity, with a proof-of-concept that checks MedQA's predictive and content validity against EHR-derived real-world cases.
Problem
Medical LLM benchmarks, largely built from medical licensing exam questions, are arbitrarily constructed and may not measure the real-world clinical capabilities they are assumed to represent, so claims based on them lack validity evidence.
Approach
Import psychometric construct validity (criterion, content, and construct validity, unified under Messick/Kane) into medical LLM benchmarking and propose empirically testing benchmarks against EHR-derived real-world clinical cases before model evaluation.
Key results
- On MedQA, GPT-4 scored 0.71 accuracy but only 0.28 on real-world EHR-matched cases, while Llama 3 scored 0.54 on MedQA and 0.48 on real-world cases, so GPT-4 tops MedQA yet ranks below Llama 3 in the real-world setting.
- Conditional accuracy alpha on real-world cases given correct MedQA answers offered little improvement over overall real-world accuracy, indicating weak predictive validity of MedQA.
- Real-world patient cases map to considerably more UMLS concepts than MedQA vignettes, and MedQA disproportionately features diagnosis over treatment questions.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MedQA accuracy to assess medical knowledge | Medical LLMs · Real-world clinical cases | accuracy: 0.54; 0.71; 0.60; 0.57; 0.49; 0.61; 0.75 | No clear effect |
| Evaluate models on real-world clinical notes rather than MedQA | GPT-4 and Llama 3 · Real-world data | accuracy: GPT-4: 0.71; 0.28; Llama 3: 0.54; 0.48 | Mixed |
| Condition real-world evaluation on correctly answered MedQA questions | Medical LLMs · Real-world data | conditional accuracy (α): 0.56; 0.29; 0.48; 0.49; 0.38; 0.49; 0.45 | No clear effect |
| Increase the number of answer choices in MedQA | Medical LLMs · MedQA | accuracy | Hurt |
| Use MedQA clinical vignettes to represent real-world clinical content | MedQA · Real-world clinical practice | content coverage | Hurt |
| Use MedQA to rank models by medical knowledge | Medical LLMs · Real-world clinical cases | model ranking: GPT-4: 0.71; Llama 3: 0.48 | Hurt |
Surprising findings
- GPT-4 ranks first on MedQA (0.71) yet falls to 0.28 on real-world cases, below Llama 3, suggesting rankings do not transfer across benchmark and real-world tasks.
- Accuracy on MedQA is higher than real-world accuracy for nearly every model, and GPT-4 had a notably high non-response rate on real-world notes.
Limitations
- Proof-of-concept experiments are limited to MedQA and a single academic medical center's EHR data.
- Matching MedQA answers to EHR notes via drug/diagnosis codes is a rough proxy for clinical similarity.
- Position paper; the proposed validation ecosystem and hospital validator model are not implemented.
Experimental setup
Datasets: MedQA MedMCQA MMLU EHR clinical notes from a large academic medical center
Benchmarks: MedQA MedMCQA MMLU EHR-matched real-world patient cases
Metrics: accuracy conditional accuracy P(correct on real-world | correct on Me model ranking agreement
Baselines: Llama 3 Chimera Llama Biomerge Orpomed JSL MedLlama PMY MedLLama
Builds on: Messick construct validity framework Cronbach and Meehl construct validity Kane validity argument
Closest papers by meaning
- MedJourney: Benchmark and Evaluation of Large Language Models over Patient Clinical Journey (NeurIPS 2024 · similarity 0.91)
- MedBench: A Large-Scale Chinese Benchmark for Evaluating Medical Large Language Models (AAAI 2024 · similarity 0.90)
- Evaluating LLMs Across Multi-Cognitive Levels: From Medical Knowledge Mastery to Scenario-Based Problem Solving (ICML 2025 · similarity 0.90)
- Measuring what Matters: Construct Validity in Large Language Model Benchmarks (NeurIPS 2025 · similarity 0.90)
- Medmarks: A Comprehensive Open-Source LLM Benchmark Suite for Medical Tasks (NeurIPS 2026 · similarity 0.90)
- Reliable and Diverse Evaluation of LLM Medical Knowledge Mastery (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.