Research map / Large language models / Agents, code and math
Evaluating LLMs Across Multi-Cognitive Levels: From Medical Knowledge Mastery to Scenario-Based Problem Solving
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MultiCogEval, a Bloom's-taxonomy-inspired three-level evaluation of LLMs on medical knowledge (MedQA/MedMCQA MCQs, reformulated mid-level tasks, and a MIMIC-IV full-path diagnosis task), finding performance drops as cognitive level rises.
Problem
Existing medical benchmarks mostly test preliminary knowledge recall via QA and do not evaluate LLM medical capabilities across multiple cognitive levels, from knowledge recall to scenario-based clinical problem solving.
Approach
Grounds a three-level evaluation framework in Bloom's Taxonomy and the medical training pipeline, aligns knowledge coverage across levels by reformulating MCQs into statement validation, multi-step rectification and answer existence tasks, links a high-level EHR-based full-path diagnosis task to the same disease pool, and normalizes accuracy for random guessing.
Key results
- DeepSeek-V3 reaches 78.3 normalized accuracy on MedQA low-level tasks but only 19.4 full-path diagnosis accuracy on the High-Level MIMIC-IV task.
- Medical-domain finetuned models gain up to ~15% over backbones on low/mid-level tasks but do not improve on the high-level task.
- DeepSeek-R1 improves mid-level performance over DeepSeek-V3 (65.5 vs 42.4 macro average), with smaller gains at high level (26.5 vs 19.4).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Evaluate models at increasingly complex medical cognitive levels | LLMs across evaluated model families · MedQA and MedMCQA, Low- and Mid-Level | normalized accuracy: significant performance drop ($\sim$20\%) | Hurt |
| Evaluate models on scenario-based full-path diagnosis | LLMs across evaluated model families · MIMIC-IV High-Level | full-path diagnosis accuracy: 19.4 | Hurt |
| Increase model parameter size | Llama and Qwen model families · MedQA, MedMCQA, and MIMIC-IV | accuracy | Helped |
| Apply medical-domain fine-tuning | medical LLMs and their backbone models · Low- and Mid-Level medical tasks | normalized accuracy: up to 15\% | Helped |
| Apply medical-domain fine-tuning | medical LLMs and their backbone models · MIMIC-IV High-Level | full-path diagnosis accuracy | Hurt |
| Add inference-time scaling | DeepSeek-V3 · Low- and Mid-Level medical tasks | normalized accuracy: +23.1 vs. +9.8 | Helped |
| Add inference-time scaling | DeepSeek-V3 and GPT-4o-mini · Low-, Mid-, and High-Level medical tasks | normalized accuracy | Helped |
| Account for examination recall in full-path diagnosis scoring | LLMs on MIMIC-IV clinical diagnosis · MIMIC-IV High-Level | diagnosis accuracy: 53.6; 19.4 | Hurt |
| Use answer-existence judgment tasks with a larger decision space | LLMs · MedQA and MedMCQA Mid-Level | normalized accuracy: over 40\% | Hurt |
| Increase cognitive difficulty across benchmark levels | Clinician validation sample · Low, Mid, and High levels | clinician accuracy: 68.8; 54.2; 23.5 | Hurt |
Surprising findings
- Medical-domain finetuning often underperforms its backbone on the high-level scenario-based diagnosis task.
- Clinicians' accuracy drops to 23.5% on high-level tasks despite strong model performance on knowledge recall.
Limitations
- Evaluation constrained by the scope of medical knowledge coverage (42 diseases) and task diversity.
- Mid-level questions were generated by GPT-4o with human verification on samples only (error rate under 5%).
- Clinician validation covers only 100 samples from four clinicians.
Experimental setup
Datasets: MedQA MedMCQA MIMIC-IV
Benchmarks: MedQA MedMCQA MIMIC-IV full-path diagnosis (MultiCogEval)
Metrics: accuracy normalized accuracy full-path diagnosis accuracy examination recall
Baselines: GPT-4o DeepSeek-V3 Llama3-70B Qwen2.5-72B Gemma2-27B Phi4-14B GPT-4o-mini DeepSeek-R1 o3-mini ClinicalCamel-70B Med42 Meditron
Builds on: Bloom's Taxonomy MedQA MedMCQA MIMIC-IV MedCAT
Closest papers by meaning
- Reliable and Diverse Evaluation of LLM Medical Knowledge Mastery (ICLR 2025 · similarity 0.90)
- MedJourney: Benchmark and Evaluation of Large Language Models over Patient Clinical Journey (NeurIPS 2024 · similarity 0.90)
- Position: Medical Large Language Model Benchmarks Should Prioritize Construct Validity (ICML 2025 · similarity 0.90)
- MedCalc-Bench: Evaluating Large Language Models for Medical Calculations (NeurIPS 2024 · similarity 0.88)
- Medmarks: A Comprehensive Open-Source LLM Benchmark Suite for Medical Tasks (NeurIPS 2026 · similarity 0.88)
- MDAgents: An Adaptive Collaboration of LLMs for Medical Decision-Making (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.