Research map / Large language models / AI and society
Intersection of Artificial Intelligence and Medical Education (Student Abstract)
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Evaluates ChatGPT-3.5 on 543 text-only pathology board-style review questions, finding 75% accuracy and categorizing five error types.
Problem
Whether a publicly available LLM can answer pathology board-style review questions for pathologists-in-training, and what kinds of errors it makes, to assess its potential role in medical education.
Approach
Applies zero-shot evaluation of ChatGPT-3.5 to a set of pathology board-style multiple-choice questions, excluding image-based items, and manually categorizes incorrect answers into error types checked against PubMed literature.
Key results
- ChatGPT-3.5 correctly answered 406 of 543 questions (75%).
- Best discipline performance was informatics & digital pathology (95%), followed by molecular pathology (91%); surgical, clinical pathology and hematopathology scored 70-73%.
- Factual inaccuracy/hallucination was the most common error type (41%), followed by misinterpreting lab values (20%) and rank-ordering errors (17%).
Surprising findings
- Hallucinations are described as the predominant error type, while the paper's own percentages are inconsistent (41% in results vs 53% in discussion).
Limitations
- Image-based questions were excluded because ChatGPT does not process images.
- Unclear how closely the question set reflects the official pathology board exam, so passing status cannot be concluded.
- Questions were drawn from online question banks; error categorization is manual and the sample is small in discipline breakdown.
Experimental setup
Datasets: pathology board-style review questions from online question banks (n=543, image questions excluded)
Benchmarks: pathology board-style review questions (543 text-only items)
Metrics: accuracy error category proportion
Closest papers by meaning
- Developing a Postgraduate Program for AI in Medicine with Kern’s Six-Step Curriculum Development Approach in Singapore (AAAI 2025 · similarity 0.85)
- Doctor-R1: Mastering Clinical Inquiry with Experiential Agentic Reinforcement Learning (ICLR 2026 · similarity 0.83)
- LLMs in the Classroom: Outcomes and Perceptions of Questions Written with the Aid of AI (AAAI 2025 · similarity 0.81)
- Simulating Viva Voce Examinations to Evaluate Clinical Reasoning in Large Language Models (NeurIPS 2025 · similarity 0.80)
- Zhongjing: Enhancing the Chinese Medical Capabilities of Large Language Model through Expert Feedback and Real-World Multi-Turn Dialogue (AAAI 2024 · similarity 0.80)
- Detecting Any instruction-to-answer interaction relationship:Universal Instruction-to-Answer Navigator for Med-VQA (ICML 2024 · similarity 0.80)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.