Research map / Large language models / Retrieval and knowledge
Reliable and Diverse Evaluation of LLM Medical Knowledge Mastery
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PretexEval, which turns knowledge-base triplets into predicate variants via equivalence/implication transformations and then renders them as text (templates plus LLM rephrasing) to evaluate LLM medical knowledge mastery on MedLAMA and DiseK.
Problem
Existing medical benchmarks may be outdated or leaked, and directly generating test samples from knowledge bases via templates or LLMs introduces factual errors and lacks structural diversity, so there is no reliable, diverse way to evaluate LLM mastery of medical knowledge bases.
Approach
Express each knowledge triplet as a predicate, derive logically equivalent or implied variants (inversion, instantiation, double negation), convert each variant into a template prototype, then rephrase with an LLM; the predicate transformations guarantee reliability while producing structural diversity. Joint accuracy measures consistency across all variants of a fact.
Key results
- On PretexEval-generated samples, GPT-4o achieves the best gain over random guessing (+31.7 MedLAMA, +26.7 DiseK), while Llama2-7B and Llama2-70B degrade by over 50% relative to LLMEval.
- Removing predicate equivalent transformation raises Llama3-70B's MedLAMA score by about 7 points (26.9 to 33.0), indicating it contributes most to diversity.
- Double negation causes the largest additional drop (around 5%), suggesting weaker handling of negated medical statements.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace direct template/LLM sample generation with PretexEval predicate-based generation | 12 evaluated LLMs · MedLAMA and DiseK | average accuracy gain over 50%: e.g., Llama2-7B 11.7 to 2.8 on DiseK vs Direct | Hurt |
| Remove predicate equivalent transformation (use only direct prototypes) (ablation) | Llama3-70B · MedLAMA | average accuracy gain over 50%: +26.9 to +33.0 when removed | Hurt |
| Remove predicate equivalent transformation (use only direct prototypes) (ablation) | Llama3-70B · DiseK | average accuracy gain over 50%: +20.9 to +27.8 when removed | Hurt |
| Remove LLM rephrasing from prototype-based generation (ablation) | Llama3-70B · MedLAMA | average accuracy gain over 50%: +26.9 to +30.4 when removed | Hurt |
| Add double negation to predicate variants (ablation) | Llama3-70B and ClinicalCamel-70B · MedLAMA | average accuracy gain over 50%: ClinCamel-70B 30.3 to 23.2 with Inversion+DN; about 5 points drop | Hurt |
| Use a different rephrasing LLM (Phi-3-medium instead of Llama3-70B) (ablation) | PretexEval generated datasets · MedLAMA and DiseK | average accuracy gain over 50%: e.g., GPT-4 +31.7 vs +32.3 on MedLAMA | No clear effect |
| Increase number of predicate variants per knowledge point (joint accuracy) | LLMs evaluated with PretexEval vs LLMEval · MedLAMA joint accuracy | joint accuracy | Hurt |
Surprising findings
- Medical-specific models ClinicalCamel-70B and Med42-70B perform similarly to Llama2-70B on original datasets but outperform it by around 7% on PretexEval samples.
- Llama3-8B slightly surpasses the 10x larger Llama2-70B on PretexEval samples.
Limitations
- Evaluation uses only two knowledge bases (MedLAMA and DiseK) and a subset of triplets; GPT-4o was evaluated on only 200 sampled triplets per dataset.
- Prototypes were hand-crafted with clinicians, so applying the framework to new knowledge bases needs manual effort.
- Answer extraction relies on keyword matching, and evaluation uses only true/false statement verification.
Experimental setup
Datasets: MedLAMA DiseK
Benchmarks: MedLAMA DiseK
Metrics: average accuracy joint accuracy
Baselines: Direct template paraphrasing LLMEval
Builds on: MedLAMA UMLS
Closest papers by meaning
- Evaluating LLMs Across Multi-Cognitive Levels: From Medical Knowledge Mastery to Scenario-Based Problem Solving (ICML 2025 · similarity 0.90)
- Position: Medical Large Language Model Benchmarks Should Prioritize Construct Validity (ICML 2025 · similarity 0.89)
- MedBench: A Large-Scale Chinese Benchmark for Evaluating Medical Large Language Models (AAAI 2024 · similarity 0.88)
- MedCalc-Bench: Evaluating Large Language Models for Medical Calculations (NeurIPS 2024 · similarity 0.87)
- Medmarks: A Comprehensive Open-Source LLM Benchmark Suite for Medical Tasks (NeurIPS 2026 · similarity 0.87)
- MedXpertQA: Benchmarking Expert-Level Medical Reasoning and Understanding (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.