Research map / Large language models / Instruction and fine-tuning
PediatricsGPT: Large Language Models as Chinese Medical Assistants for Pediatric Applications
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds PedCorpus (300k+ Chinese pediatric instructions) and trains PediatricsGPT on Baichuan2 via a four-stage pipeline (hybrid instruction CPT, full-parameter SFT, DFPO preference optimization, and LoRA mixture of universal-specific experts), outperforming Chinese medical LLMs on pediatric and public benchmarks.
Problem
Chinese LLMs perform poorly on pediatric consultation due to inadequate domain instruction data and fragile training procedures (knowledge inconsistency, role-playing-heavy SFT, unstable RLHF).
Approach
Combines a multi-source pediatric instruction dataset with a staged training pipeline: hybrid instruction pre-training (converting Input-Output instructions into completion-style plain text) during CPT, a single-stage direct following preference optimization (DFPO), and a LoRA mixture with one universal expert plus several specific experts for parameter-efficient SFT.
Key results
- PediatricsGPT-13B achieves the best ROUGE-1 (60.85), BLEU-4 (59.34) and GLEU (57.22) on MedKQ&A, surpassing GPT-4 on most metrics.
- In GPT-4 pairwise evaluation, the full PediatricsGPT-13B model wins 68% vs ChatGPT on MedKQ&A and 54% on EviDiag.
- In doctor evaluation, the full model reaches 56% win rate on MedKQ&A and 38% on TreRecom against ChatGPT.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use PediatricsGPT-13B instead of GPT-4 | MedKQ&A · MedKQ&A | GLEU: 53.72 -> 57.22 | Helped |
| Use PediatricsGPT-13B instead of GPT-4 | EviDiag · EviDiag | GLEU: 36.79 -> 37.15 | Helped |
| Use PediatricsGPT-13B instead of GPT-4 | TreRecom · TreRecom | GLEU: 55.26 -> 54.65 | Hurt |
| Add continuous pre-training (ablation) | PediatricsGPT · MedKQ&A | GPT-4 win rate: 61% -> 68% | Helped |
| Add continuous pre-training (ablation) | PediatricsGPT · EviDiag | GPT-4 win rate: 46% -> 54% | Helped |
| Add hybrid instruction pre-training (ablation) | PediatricsGPT · MedKQ&A | GPT-4 win rate: 67% -> 68% | Helped |
| Add hybrid instruction pre-training (ablation) | PediatricsGPT · EviDiag | GPT-4 win rate: 52% -> 54% | Helped |
| Add full-parameter SFT (ablation) | PediatricsGPT · MedKQ&A | GPT-4 win rate: 65% -> 68% | Helped |
| Add parameter-efficient SFT (ablation) | PediatricsGPT · MedKQ&A | GPT-4 win rate: 63% -> 68% | Helped |
| Use MUE instead of vanilla single LoRA (ablation) | PediatricsGPT · EviDiag | GPT-4 win rate: 52% -> 54% | Helped |
| Use MUE instead of vanilla single LoRA (ablation) | PediatricsGPT · TreRecom | GPT-4 win rate: 56% -> 58% | Helped |
| Add universal expert (ablation) | PediatricsGPT · CMD | GPT-4 win rate: 44% -> 46% | Helped |
Surprising findings
- Removing the full-parameter SFT or the parameter-efficient SFT is reported to hurt results less uniformly than expected; the single-LoRA removal affects pediatric tasks more than general benchmarks.
- Vanilla DPO and RLHF are reported as inferior to DFPO in doctor evaluation, while the paper notes no error bars are reported.
Limitations
- Security risks when deployed online (manipulation attacks generating unsafe content).
- Does not support languages other than Chinese.
- No error bars or statistical significance reported due to LLM training cost.
Experimental setup
Datasets: PedCorpus PedCorpus-CPT PedCorpus-DFPO Alpaca ShareGPT Huatuo-26M MedDialog CMeKG Chinese Wikipedia webMedQA CMD
Benchmarks: MedKQ&A EviDiag TreRecom CMD webMedQA
Metrics: ROUGE-1 ROUGE-2 ROUGE-L BLEU-1 BLEU-2 BLEU-3 BLEU-4 GLEU Distinct-1 Distinct-2
Baselines: Baichuan2-7B Baichuan2-13B HuatuoGPT DISC-MedLLM Zhongjing HuatuoGPT-II Meditron-7B Llama3.1-8B ChatGPT GPT-4
Builds on: DPO LoRA Baichuan2 Mixture of experts
Closest papers by meaning
- MedBench: A Large-Scale Chinese Benchmark for Evaluating Medical Large Language Models (AAAI 2024 · similarity 0.88)
- Zhongjing: Enhancing the Chinese Medical Capabilities of Large Language Model through Expert Feedback and Real-World Multi-Turn Dialogue (AAAI 2024 · similarity 0.88)
- CareBot: A Pioneering Full-Process Open-Source Medical Language Model (AAAI 2025 · similarity 0.87)
- MedJourney: Benchmark and Evaluation of Large Language Models over Patient Clinical Journey (NeurIPS 2024 · similarity 0.86)
- Efficiently Democratizing Medical LLMs for 50 Languages via a Mixture of Language Family Experts (ICLR 2025 · similarity 0.85)
- Instruction Tuning Large Language Models to Understand Electronic Health Records (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.