Research map / Large language models / Language, safety and interpretability
CUPCase: Clinically Uncommon Patient Cases and Diagnoses Dataset
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds CUPCase, a 3,562-case benchmark from BMC case reports with multiple-choice and open-ended diagnosis tasks, and evaluates general-purpose and clinical LLMs, finding GPT-4o best.
Problem
Existing medical benchmarks rely on exam questions or textbook/literature cases and lack the complexity of real-world uncommon patient cases (rare diseases, atypical presentations, unexpected treatment responses).
Approach
Builds a benchmark from real-world BMC clinical case reports, removing final-diagnosis mentions with GPT-4o-mini, generating multiple-choice distractors via embedding similarity to ICD-10-CM codes, and evaluating with partial case-text inputs.
Key results
- GPT-4o achieved the best multiple-choice accuracy of 87.9% and free-text BERTScore F1 of 76.42, outperforming clinical LLMs such as Meditron-70B (85.55% accuracy) and MedLM-Large (71.32 BERTScore F1).
- GPT-4o retained about 87-88% of its full-text performance using only the first 20% of case tokens.
- Meditron-70B scored 85.55% accuracy but only 59.77 BERTScore F1, showing weak agreement between the two tasks.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use GPT-4o instead of Llama3-70B Instruct | CUPCase multiple-choice diagnosis · CUPCase multiple-choice | accuracy: 85.70 -> 87.90 | Helped |
| Use GPT-4o instead of Llama3-70B Instruct | CUPCase free-text diagnosis · CUPCase free-text diagnosis | BERTScore F1: 72.67 -> 76.42 | Helped |
| Use Meditron-70B instead of MedLM-Large | CUPCase diagnosis · CUPCase multiple-choice | accuracy: 84.40 -> 85.55 | Helped |
| Use Meditron-70B instead of MedLM-Large | CUPCase diagnosis · CUPCase free-text diagnosis | BERTScore F1: 71.32 -> 59.77 | Hurt |
| Continue medical-domain training with Llama3 OpenBioLLM 70B instead of using Llama3-70B Instruct | CUPCase diagnosis · CUPCase multiple-choice | accuracy: 85.70 -> 84.27 | Hurt |
| Use Llama3.1-8B Instruct instead of Llama3-8B Instruct | CUPCase diagnosis · CUPCase free-text diagnosis | BERTScore F1: 50.69 -> 66.97 | Helped |
| Provide more case-presentation tokens | GPT-4o · CUPCase diagnosis across cumulative token percentages | accuracy | Helped |
| Use only the first 20% of case-presentation tokens | GPT-4o · CUPCase multiple-choice | accuracy: 76.8% ± 3.46 | Hurt |
| Use only the first 20% of case-presentation tokens | GPT-4o · CUPCase free-text diagnosis | BERTScore F1: 67.74% ± .5 | Hurt |
Surprising findings
- General-purpose models such as GPT-4o and Llama3-70B Instruct outperformed clinical LLMs fine-tuned for the medical domain.
- Llama3.1-8B Instruct reached 66.97 BERTScore F1 versus 50.69 for Llama3-8B Instruct, a large gap between closely related models.
Limitations
- Dataset drawn only from BMC Journal of Medical Case Reports, so may not represent the full clinical domain.
- Only zero-shot evaluation and text-only case presentations were used; figures were excluded.
- Cannot rule out that case reports were in the evaluated models' training data; only ~1% of preprocessing was manually validated.
Experimental setup
Datasets: BMC Journal of Medical Case Reports case presentations (2012-2020) ICD-10-CM diagnosis descriptions
Benchmarks: CUPCase
Metrics: accuracy BERTScore F1 standard deviation over bootstrap samples
Baselines: Llama3-70B Instruct Llama3.1-70B Instruct Llama3.1-8B Instruct Llama3-8B Instruct Meditron-70B Meditron-7B MedLM-Large Llama3 OpenBioLLM 70B Llama3 OpenBioLLM 8B BioMistral-7B-DARE BioMedGPT-LM-7B Meerkat
Builds on: lm-evaluation-harness BERTScore
Closest papers by meaning
- DiReCT: Diagnostic Reasoning for Clinical Notes via Large Language Models (NeurIPS 2024 · similarity 0.87)
- MedJourney: Benchmark and Evaluation of Large Language Models over Patient Clinical Journey (NeurIPS 2024 · similarity 0.87)
- MedCalc-Bench: Evaluating Large Language Models for Medical Calculations (NeurIPS 2024 · similarity 0.87)
- MedBench: A Large-Scale Chinese Benchmark for Evaluating Medical Large Language Models (AAAI 2024 · similarity 0.87)
- EHRNoteQA: An LLM Benchmark for Real-World Clinical Practice Using Discharge Summaries (NeurIPS 2024 · similarity 0.86)
- Medmarks: A Comprehensive Open-Source LLM Benchmark Suite for Medical Tasks (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.