Research map / Large language models / Alignment and preferences
UltraMedical: Building Specialized Generalists in Biomedicine
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds UltraMedical, a ~410K-instruction biomedical dataset with ~100K GPT-4-ranked preference annotations, and uses it to fine-tune Llama-3 8B/70B models and train medical and general reward models.
Problem
Open-source LLMs lag proprietary models in biomedicine, and preference learning (RLHF/RLAIF, DPO, KTO) is under-explored in medicine because of scarce specialized instruction and preference data.
Approach
A data-centric collection that mixes manual and synthetic medical instructions (MedQA-Evol, TextBookQA, WikiInstruct) with multi-model completions ranked by GPT-4, then applies SFT, offline DPO/KTO, reward modeling, and one round of on-policy iterative preference learning with best-of-N reranking on Llama-3 models.
Key results
- UltraMix Best (Ens) with Llama-3-70B reaches 86.49 average on medical MCQ benchmarks and 85.4 on MedQA, the highest reported among open-source models in the paper.
- UltraMix Best (Ens) with Llama-3-8B reaches 78.32 average, above Llama-3-8B-Instruct (68.56) and OpenBioLLM-8B (72.48) on the same medical benchmarks.
- UltraMedRM-8B achieves 80.53 average on the Medical RewardBench and 85.53 on RewardBench, versus 87.13 for PairRM-LLaMA3-8B on RewardBench.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add general-domain data to medical-only SFT (ablation) | Llama-3-8B UltraMedical · Medical benchmarks | average score: 75.20 -> 75.90 | Helped |
| add general-domain data to medical-only SFT (ablation) | Llama-3-70B UltraMedical · Medical benchmarks | average score: 84.62 -> 84.27 | Hurt |
| apply Vanilla DPO after medical-only SFT (ablation) | Llama-3-8B UltraMedical · Medical benchmarks | average score: 75.20 -> 76.61 | Helped |
| apply Vanilla KTO after medical-only SFT (ablation) | Llama-3-8B UltraMedical · Medical benchmarks | average score: 75.20 -> 76.61 | Helped |
| replace medical-only preference data with UltraMix preference data for Vanilla DPO (ablation) | Llama-3-8B UltraMedical · Medical benchmarks | average score: 76.61 -> 77.29 | Helped |
| replace medical-only preference data with UltraMix preference data for Vanilla KTO (ablation) | Llama-3-8B UltraMedical · Medical benchmarks | average score: 76.61 -> 76.74 | Helped |
| use reward-model-labeled on-policy preferences with iterative DPO instead of Vanilla DPO (ablation) | Llama-3-8B UltraMix · Medical benchmarks | average score: 77.29 -> 77.51 | Helped |
| use reward-model-labeled on-policy preferences with iterative KTO instead of Vanilla KTO (ablation) | Llama-3-8B UltraMix · Medical benchmarks | average score: 76.74 -> 77.41 | Helped |
| replace medical-only preference data with UltraMix preference data for Vanilla DPO (ablation) | Llama-3-70B UltraMedical · Medical benchmarks | average score: 85.57 -> 85.56 | No clear effect |
| use reward-model sorting instead of self-consistency sorting | Llama-3-8B UltraMedical · Medical benchmarks | average score: 78.33 -> 78.67; 78.40 -> 79.52; 78.28 -> 78.31; 77.98 -> 77.21 | Mixed |
| use reward-model sorting instead of self-consistency sorting | Llama-3-70B UltraMedical · Medical benchmarks | average score: 86.48 -> 85.61; 86.92 -> 85.30; 86.41 -> 86.27; 86.11 -> 85.62 | Mixed |
Surprising findings
- Reward models outperform self-consistency ensembles for 8B models in reranking but are less effective than self-consistency for 70B models.
- Preference learning gains from offline completions were modest, especially for the 70B model, which the authors attribute to distribution mismatch.
Limitations
- Relies on GPT-4 annotations for preferences, which may introduce bias and self-preference.
- Iterative preference learning was run for only one round due to limited computational resources.
- Reward models underperform on the hard, human, and length subsets of the medical reward benchmark.
Experimental setup
Datasets: MedQA MedMCQA PubMedQA MedQuAD MedInstruct-52k Medical-Instruction-120k ChatDoctor UltraChat Open-Orca Evol-Instruct-v2 UltraFeedback UltraSafety
Benchmarks: MedQA MedMCQA PubMedQA MMLU medical subsets K-QA MT-Bench AlpacaEval 2 MMLU GPQA GSM8K RewardBench Medical RewardBench
Metrics: accuracy average score GPT-4 judged completeness and hallucination length-controlled win rate win rate reward model pairwise accuracy
Baselines: Llama-3-8B-Instruct Llama-3-70B-Instruct OpenBioLLM-8B OpenBioLLM-70B Med42-70B BioMistral-7B Meerkat-7B Internist-7B Med-PaLM 2 GPT-4 (Medprompt) UltraRM-13b Eurus-RM-7B
Builds on: DPO KTO Zephyr UltraFeedback Medprompt
Closest papers by meaning
- Why Specialist Models Still Matter: A Heterogeneous Multi-Agent Paradigm for Medical Artificial Intelligence (ICML 2026 · similarity 0.86)
- Zhongjing: Enhancing the Chinese Medical Capabilities of Large Language Model through Expert Feedback and Real-World Multi-Turn Dialogue (AAAI 2024 · similarity 0.84)
- MDAgents: An Adaptive Collaboration of LLMs for Medical Decision-Making (NeurIPS 2024 · similarity 0.84)
- CareBot: A Pioneering Full-Process Open-Source Medical Language Model (AAAI 2025 · similarity 0.83)
- Efficiently Democratizing Medical LLMs for 50 Languages via a Mixture of Language Family Experts (ICLR 2025 · similarity 0.83)
- Tag-LLM: Repurposing General-Purpose LLMs for Specialized Domains (ICML 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.