Research map / Large language models / AI and society
IMQC: A Large Language Model Platform for Medical Quality Control
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Reports IMQC, an LLM platform (EMR-LLM, continually pre-trained PULSE-20B then LoRA-tuned on EMR instruction data with curriculum ordering) that converts medical quality control indicators into a structured representation and computes them from EMRs, deployed in Shanghai.
Problem
Calculating medical quality control indicators from electronic medical records relies on expert-written heuristic rules, which is slow, hard to generalize across indicators, and labor-intensive.
Approach
Continually pre-train an open LLM on medical corpora, instruction-tune it on EMR-derived tasks using a curriculum ordered by difficulty, and use it to convert indicators into a representation language (definition, formula, facts, logical rules, EMR fields, knowledge) and compute indicators via least-to-most decomposition, fact extraction, and logical rule integration.
Key results
- EMR-LLM with its calculation method reaches 93.31% accuracy and 92.15 F1 on the QCIC test set, versus 88.53% accuracy for the best baseline (EMR-LLM with ICL).
- On zero-shot EMR tasks, EMR-LLM scores 85.58 QCIC, 90.12 EMR ID, 92.85 CSD, 89.28 LCC, 96.91 TS accuracy, beating PULSE-20B on all tasks.
- Deployed system processes about 1,000 EMRs per hour on one H800 GPU versus about 10 per hour for manual review, and has handled 1,245 indicators since February 2024.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use the proposed quality-control indicator calculation method instead of ICL inference (ablation) | EMR-LLM · QCIC test set | accuracy: 88.53 -> 93.31; improvement of 4.78% | Helped |
| use the proposed quality-control indicator calculation method instead of ICL inference (ablation) | EMR-LLM · QCIC test set | F1 score: 86.91 -> 92.15; improvement of 5.24% | Helped |
| continually pre-train and instruction-tune the model for EMR tasks | PULSE-20B · QCIC (ICL) | accuracy: 84.38 -> 88.53; 4.15% higher | Helped |
Surprising findings
- Fine-tuned BERT (74.31 accuracy) and several open LLMs with ICL scored well below PULSE-20B with ICL, showing base model choice matters a lot.
- The authors report that manual expert calculation accuracy often exceeds 95%, slightly above the platform's 93.31%.
Limitations
- Test set and EMR data are private (de-identified samples only on request), limiting reproducibility.
- Platform accuracy (93.31%) is slightly below expert manual accuracy (often above 95%).
- Current deployment covers Shanghai secondary and tertiary hospitals; generalization to other regions and EMR writing styles is future work.
Experimental setup
Datasets: Medical books Clinical guidelines Expert consensus Indicator compilations CPT MIMIC PubMed Wikipedia falcon-refinedweb WuDao CLUE CBLUE
Benchmarks: Manually verified QCIC test set (108 indicators, 42 diseases) EMR ID/CSD/LCC/TS constructed tasks
Metrics: accuracy F1 score
Baselines: BERT ChatGLM3-6B Mistral-7B-Instruct Llama3-8B-Instruct Huatuo2-7B PULSE-20B Zero-shot ICL CoT Least-to-Most Self-Consistency
Builds on: PULSE LoRA least-to-most prompting chain-of-thought curriculum learning
Closest papers by meaning
- MedBench: A Large-Scale Chinese Benchmark for Evaluating Medical Large Language Models (AAAI 2024 · similarity 0.82)
- MedJourney: Benchmark and Evaluation of Large Language Models over Patient Clinical Journey (NeurIPS 2024 · similarity 0.82)
- MedCalc-Bench: Evaluating Large Language Models for Medical Calculations (NeurIPS 2024 · similarity 0.81)
- MedSafetyBench: Evaluating and Improving the Medical Safety of Large Language Models (NeurIPS 2024 · similarity 0.81)
- Instruction Tuning Large Language Models to Understand Electronic Health Records (NeurIPS 2024 · similarity 0.81)
- Hierarchical Divide-and-Conquer for Fine-Grained Alignment in LLM-Based Medical Evaluation (AAAI 2025 · similarity 0.80)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.