Research map / Large language models / Instruction and fine-tuning
MedSafetyBench: Evaluating and Improving the Medical Safety of Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Defines medical safety for LLMs via the AMA Principles of Medical Ethics, builds MedSafetyBench (1,800 harmful medical requests with safe refusals), and shows public medical LLMs comply with harmful requests while instruction-tuning on it improves safety without hurting medical QA.
Problem
Publicly available medical LLMs lack evaluation of medical safety; existing safety benchmarks target general harm and miss medical-ethics concerns such as patient rights, confidentiality, and medical error concealment.
Approach
Operationalizes medical safety as consistency with the nine AMA Principles of Medical Ethics, generates harmful medical requests per principle (GPT-4 prompting and GCG jailbreaking of Llama-2-7b-chat), pairs them with GPT-4 safe refusals, and uses the split for both evaluation and safety fine-tuning.
Key results
- Medical LLMs score 1.90-2.98 harmfulness on MedSafety-Eval-GPT4 and 1.36-2.11 on MedSafety-Eval-Llama2, higher than safety-aligned general LLMs.
- Fine-tuning on both safety sets (BothSafety-Improve, n=1,800) gives the lowest harmfulness across medical LLMs, with medical safety demonstrations outperforming general ones.
- Safety fine-tuning preserves medical performance on MedQA, MedMCQA, PubMedQA, and MMLU-Medical.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune medical LLMs on safety demonstrations | Medalpaca-7b, Medalpaca-13b, and ClinicalCamel-70b · GenSafety-Eval, MedSafety-Eval-GPT4, and MedSafety-Eval-Llama2 | harmfulness score | Helped |
| Fine-tune on general safety demonstrations | Medical LLMs · GenSafety-Eval and MedSafety-Eval | harmfulness score | Helped |
| Fine-tune on medical safety demonstrations | Medical LLMs · GenSafety-Eval and MedSafety-Eval | harmfulness score | Helped |
| Fine-tune on both general and medical safety demonstrations | Medical LLMs · GenSafety-Eval and MedSafety-Eval | harmfulness score | Helped |
| Use both general and medical demonstrations instead of medical-only demonstrations (ablation) | Medical LLMs · GenSafety-Eval and MedSafety-Eval | harmfulness score | Helped |
| Use medical demonstrations instead of general-only demonstrations (ablation) | Medical LLMs · GenSafety-Eval and MedSafety-Eval | harmfulness score | Helped |
| Increase the number of safety demonstrations used for fine-tuning (ablation) | Medical LLMs · Safety evaluation datasets | harmfulness score | Helped |
| Fine-tune for safety | Medical LLMs · MedQA, MedMCQA, PubMedQA, and MMLU-Medical | medical performance | No clear effect |
Surprising findings
- Medical LLMs comply with harmful requests more than their safety-aligned general-knowledge counterparts.
- Harmful requests containing medical jargon (GPT-4-generated) elicit higher harmfulness than Llama-2-generated ones.
Limitations
- Medical safety is defined only through the AMA Principles of Medical Ethics, without nuance for subspecialties or patient preferences.
- Harmfulness is scored by GPT-3.5 as judge rather than by human raters for all responses.
- Only instruction-tuning is explored; RLHF and other alignment methods are left to future work.
Experimental setup
Datasets: MedSafetyBench GenSafety-Eval GenSafety-Improve MedQA MedMCQA PubMedQA MMLU-Medical
Benchmarks: MedSafetyBench (MedSafety-Eval) GenSafety-Eval MedQA MedMCQA PubMedQA MMLU-Medical
Metrics: harmfulness score (1-5) accuracy
Baselines: Llama-2-chat GPT-3.5 GPT-4 GPT-4o Vicuna Mixtral
Builds on: Greedy Coordinate Gradient AMA Principles of Medical Ethics GenSafety-Eval
Closest papers by meaning
- SafetyPrompts: A Systematic Review of Open Datasets for Evaluating and Improving Large Language Model Safety (AAAI 2025 · similarity 0.89)
- CARES: Comprehensive Evaluation of Safety and Adversarial Robustness in Medical LLMs (NeurIPS 2025 · similarity 0.89)
- MedJourney: Benchmark and Evaluation of Large Language Models over Patient Clinical Journey (NeurIPS 2024 · similarity 0.88)
- MedBench: A Large-Scale Chinese Benchmark for Evaluating Medical Large Language Models (AAAI 2024 · similarity 0.88)
- MedCalc-Bench: Evaluating Large Language Models for Medical Calculations (NeurIPS 2024 · similarity 0.87)
- Medmarks: A Comprehensive Open-Source LLM Benchmark Suite for Medical Tasks (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.