Research map / Large language models / Instruction and fine-tuning
CareBot: A Pioneering Full-Process Open-Source Medical Language Model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds CareBot, a bilingual medical LLM on LLaMA3-8B via two-stage continual pre-training (Stable and Boost CPT), a DataRater quality filter, ConFilter multi-turn SFT selection, and DPO alignment.
Problem
Open-source LLMs underperform in specialized medical domains; existing medical LLMs rely mainly on SFT or lossy pipelines (synthetic pre-training data, multi-stage base model transformation) that cause forgetting and carry data-quality problems.
Approach
A two-stage continual pre-training schedule (Stable CPT with 19:1 medical-to-general mix, then Boost CPT mixing medical pre-training data with medical SFT data) combined with a bge-m3-based quality rater for CPT data and a cross-entropy-ratio score (ConFilter) for selecting multi-turn dialogues, followed by SFT and DPO.
Key results
- On the five-benchmark medical knowledge table, CareBot-Chat averages 61.58 vs. 52.93 for ChatGPT and 54.89 for HuatuoGPT II-7B.
- On CMtMedQA, CareBot-Chat (RL) reaches an average of 4.58 vs. 4.03 for HuatuoGPT II-7B and 4.29 for ChatGPT on the 1-5 GPT-4-judged rubric.
- On CMB-Clin, CareBot-Chat (RL) averages 4.31 vs. 3.96 for HuatuoGPT II-7B.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use two-stage CPT with Stable CPT followed by Boost CPT instead of continuing Stable CPT (ablation) | CareBot CPT · Chinese medical benchmarks | accuracy | Helped |
| Continue Stable CPT for 20B additional tokens rather than add Boost CPT (ablation) | CareBot CPT · Medical benchmarks | accuracy | No clear effect |
| Select multi-turn SFT dialogues with ConFilter instead of random selection (ablation) | CareBot · CMtMedQA | relevance | Helped |
| Select multi-turn SFT dialogues with ConFilter instead of random selection (ablation) | CareBot · CMtMedQA | fluency | No clear effect |
| Add DPO alignment after SFT (ablation) | CareBot-Chat · CMtMedQA | average score: 4.53 -> 4.58 | Helped |
| Add DPO alignment after SFT (ablation) | CareBot-Chat · CMB-Clin | average score: 4.22 -> 4.31 | Helped |
| Add DPO alignment after SFT (ablation) | CareBot-Chat · Medical concept knowledge average | score: 61.58 -> 61.39 | Hurt |
| Use CareBot-Chat rather than HuatuoGPT ll | CareBot-Chat · CMtMedQA Turn 1 | average score: 4.77 -> 4.52 (-0.25) | Hurt |
| Use CareBot-Chat rather than HuatuoGPT ll | CareBot-Chat · CMtMedQA Turns 2–10 | average score: 4.17 -> 4.47 (+0.30); 3.73 -> 4.50 (+0.77); 3.66 -> 4.63 (+0.97); 3.63 -> 4.63 (+1.00); 3.65 -> 4.68 (+1.03); 3.79 -> 4. | Helped |
| Use two-stage CPT rather than standard CPT (ablation) | CareBot CPT · English medical benchmarks | accuracy | Hurt |
Surprising findings
- CareBot-Chat trails HuatuoGPT II-7B on single-turn HuatuoEval comparisons, and the authors attribute part of this to hallucinations in the judged HuatuoGPT II response.
- Boost CPT with 20B extra tokens under Stable CPT alone showed minimal gains, while the Boost CPT stage gave a marked improvement.
Limitations
- Built on LLaMA3-8B, whose English strength leads to only competitive Chinese results against HuatuoGPT II on some Chinese benchmarks (CMB, CMMLU-Med, C-Eval-Med).
- Evaluation of consultation relies on GPT-4 judging and pairwise HuatuoEval comparisons, which the authors show can be inconsistent.
- Several method details (DataRater thresholds, ConFilter threshold, data sizes) are described only qualitatively.
Experimental setup
Datasets: C4 Pile Wudao PubMed Chinese Medical Dialogue Data Huatuo26M MedDialog ChatMed Consult Dataset ChatDoctor CMB MedQA CMtMedQA
Benchmarks: MedQA MMLU-Med CMB CMMLU-Med C-Eval-Med MedMCQA PubMedQA Huatuo26M-test CMtMedQA CMB-Clin
Metrics: accuracy fluency, relevance, completeness, proficiency scores (1-5) pairwise win rate (HuatuoEval)
Baselines: ChatGPT HuatuoGPT-7B HuatuoGPT II-7B Zhongjing-13B MedAlpaca-7B BioMistral-7B
Builds on: LLaMA3 Deita DPO
Closest papers by meaning
- Zhongjing: Enhancing the Chinese Medical Capabilities of Large Language Model through Expert Feedback and Real-World Multi-Turn Dialogue (AAAI 2024 · similarity 0.91)
- MedJourney: Benchmark and Evaluation of Large Language Models over Patient Clinical Journey (NeurIPS 2024 · similarity 0.88)
- MedBench: A Large-Scale Chinese Benchmark for Evaluating Medical Large Language Models (AAAI 2024 · similarity 0.87)
- PediatricsGPT: Large Language Models as Chinese Medical Assistants for Pediatric Applications (NeurIPS 2024 · similarity 0.87)
- Medmarks: A Comprehensive Open-Source LLM Benchmark Suite for Medical Tasks (NeurIPS 2026 · similarity 0.87)
- Doctor-R1: Mastering Clinical Inquiry with Experiential Agentic Reinforcement Learning (ICLR 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.