Research map / Large language models / Instruction and fine-tuning
Zhongjing: Enhancing the Chinese Medical Capabilities of Large Language Model through Expert Feedback and Real-World Multi-Turn Dialogue
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds Zhongjing, a Chinese medical LLaMA-based model trained through continual pre-training, SFT on a new multi-turn dialogue dataset (CMtMedQA), and RLHF with expert rankings, evaluated against Chinese medical LLMs and ChatGPT via GPT-4 and human experts.
Problem
Chinese medical LLMs rely on single-turn, distilled SFT data and lack proactive doctor-like inquiry, multi-turn comprehension, and alignment with expert intent.
Approach
A full pipeline (continual pre-training on real medical corpora, SFT with a multi-turn real doctor-patient dataset including proactive inquiries, then RLHF with a reward model trained on expert rankings and PPO) applied to Ziya-LLaMA-13B, plus a nine-ability annotation and evaluation rubric for medical dialogue.
Key results
- Zhongjing outperforms HuatuoGPT across safety, professionalism, and fluency dimensions in both single-turn and multi-turn evaluation, per the paper's claims.
- Zhongjing is reported to match ChatGPT in some abilities despite roughly 1% of its parameters, but falls behind ChatGPT in safety.
- Ablations show continual pre-training mainly improves professionalism and RLHF mainly improves safety and response length.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Zhongjing instead of HuatuoGPT | Chinese medical dialogue models · CMtMedQA-test and Huatuo26M-test | Safety, professionalism, and fluency | Helped |
| Use Zhongjing instead of other baseline models | Chinese medical dialogue models · CMtMedQA-test | Professionalism and fluency | Helped |
| Use Zhongjing instead of Ziya-LLaMA | Ziya-LLaMA · Chinese medical dialogue | Medical capabilities | Helped |
| Use Zhongjing instead of ChatGPT | ChatGPT · Chinese medical dialogue | Safety | Hurt |
| Add continuous pre-training on medical corpus (ablation) | Zhongjing · CMtMedQA-test | Professionalism | Helped |
| Add continuous pre-training on medical corpus (ablation) | Zhongjing · CMtMedQA-test | Medical capabilities | Helped |
| Add RLHF (ablation) | Zhongjing · CMtMedQA-test | Safety | Helped |
| Add RLHF (ablation) | Zhongjing · CMtMedQA-test | Response length | Helped |
| Add RLHF (ablation) | Zhongjing · CMtMedQA-test | Fluency | Helped |
| Add RLHF (ablation) | Zhongjing · CMtMedQA-test | Professionalism | Helped |
Surprising findings
- Distilled self-instruct data (as in DoctorGLM) led to poorer performance than the proposed model with similar size and instruction scale.
- Zhongjing falls behind ChatGPT in safety despite its alignment stages.
Limitations
- Cannot guarantee accuracy of responses; hallucination remains an issue.
- Falls behind much larger models such as ChatGPT in safety, suggesting scale limits.
- Evaluation relies partly on GPT-4 judging and a limited set of comparisons.
Experimental setup
Datasets: CMtMedQA ChatMed Medical Books Medical Wiki CMeKG Huatuo26M Hospital Data
Benchmarks: CMtMedQA-test Huatuo26M-test
Metrics: win rate tie rate loss rate
Baselines: ChatGPT Ziya-LLaMA BenTsao DoctorGLM HuatuoGPT
Builds on: LLaMA Ziya-LLaMA PPO self-instruct LoRA
Closest papers by meaning
- CareBot: A Pioneering Full-Process Open-Source Medical Language Model (AAAI 2025 · similarity 0.91)
- MedBench: A Large-Scale Chinese Benchmark for Evaluating Medical Large Language Models (AAAI 2024 · similarity 0.90)
- TCM-Ladder: A Benchmark for Multimodal Question Answering on Traditional Chinese Medicine (NeurIPS 2025 · similarity 0.88)
- Doctor-R1: Mastering Clinical Inquiry with Experiential Agentic Reinforcement Learning (ICLR 2026 · similarity 0.88)
- PediatricsGPT: Large Language Models as Chinese Medical Assistants for Pediatric Applications (NeurIPS 2024 · similarity 0.88)
- Chiron-o1: Igniting Multimodal Large Language Models towards Generalizable Medical Reasoning via Mentor-Intern Collaborative Search (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.