Research map / Large language models / Retrieval and knowledge
MedCalc-Bench: Evaluating Large Language Models for Medical Calculations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces MedCalc-Bench, a 1047-instance benchmark of 55 medical calculator tasks with patient notes, ground-truth answers and explanations, and evaluates LLMs under zero-shot, zero-shot CoT, and one-shot CoT prompting.
Problem
LLMs are assessed mainly on knowledge-based multiple-choice QA, with no benchmark for quantitative medical calculation (clinical calculators requiring knowledge of equations/rules, attribute extraction from long notes, and arithmetic).
Approach
A manually reviewed, open-ended benchmark built from popular MDCalc calculators, with patient notes drawn from Open-Patients (PMC-Patients, MedQA, TREC) or synthesized, and template-based step-by-step explanations; error analysis splits failures into knowledge, extraction, computation, and other errors.
Key results
- GPT-4 with one-shot CoT achieved the best average accuracy of 50.91% on MedCalc-Bench.
- Zero-shot direct prompting gave GPT-4 only 20.82% average accuracy, improving to 37.92% with zero-shot CoT.
- Llama 3-70B with zero-shot CoT reached 35.53% average, close to GPT-4 zero-shot CoT (37.92%), while MediTron-70B and PMC-LLaMA-13B scored near zero zero-shot.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add zero-shot CoT prompting instead of zero-shot direct prompting | GPT-4 · MedCalc-Bench overall | accuracy (%): 20.82% -> 37.92% | Helped |
| Add a one-shot calculation exemplar to zero-shot CoT prompting | GPT-4 · MedCalc-Bench overall | accuracy (%): 37.92% -> 50.91% | Helped |
| Add zero-shot CoT prompting instead of zero-shot direct prompting | GPT-4 · MedCalc-Bench Physical | accuracy (%): +36.67% | Helped |
| Add a one-shot exemplar to zero-shot CoT prompting | GPT-4 · MedCalc-Bench Physical | accuracy (%): +6.25% | Helped |
| Add zero-shot CoT prompting instead of zero-shot direct prompting | GPT-4 · MedCalc-Bench Diagnosis | accuracy (%): +8.33% | Helped |
| Add a one-shot exemplar to zero-shot CoT prompting | GPT-4 · MedCalc-Bench Diagnosis | accuracy (%): +25% | Helped |
Surprising findings
- Under zero-shot CoT, most GPT-4 errors (about 35% of total errors by type, 57% of error distribution) are knowledge errors, but one-shot exemplars shift errors toward extraction and computation.
- Medical-domain models PMC-LLaMA-13B and MediTron-70B perform poorly in zero-shot settings, with MediTron-70B underperforming Mistral-7B, attributed to lack of instruction tuning.
Limitations
- Dataset is limited in size (1047 instances) due to costly manual verification.
- Ambiguity in extracting the correct entity when a note mentions values multiple times or describes multiple visits.
- Only zero-shot and one-shot settings evaluated; few-shot not explored.
Experimental setup
Datasets: Open-Patients PMC-Patients MedQA-USMLE TREC Clinical Decision Support TREC Clinical Trials
Benchmarks: MedCalc-Bench
Metrics: accuracy
Baselines: PMC-LLaMA-13B MediTron-70B Mistral-7B Mixtral-8x7B Llama 3-8B Llama 3-70B GPT-3.5 GPT-4
Builds on: chain-of-thought prompting PMC-Patients Open-Patients
Closest papers by meaning
- Medmarks: A Comprehensive Open-Source LLM Benchmark Suite for Medical Tasks (NeurIPS 2026 · similarity 0.92)
- MedJourney: Benchmark and Evaluation of Large Language Models over Patient Clinical Journey (NeurIPS 2024 · similarity 0.91)
- MedBench: A Large-Scale Chinese Benchmark for Evaluating Medical Large Language Models (AAAI 2024 · similarity 0.90)
- Position: Medical Large Language Model Benchmarks Should Prioritize Construct Validity (ICML 2025 · similarity 0.89)
- Hierarchical Divide-and-Conquer for Fine-Grained Alignment in LLM-Based Medical Evaluation (AAAI 2025 · similarity 0.89)
- MedXpertQA: Benchmarking Expert-Level Medical Reasoning and Understanding (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.