Research map / Large language models / Retrieval and knowledge
ClinicalRAG: Automating Pharmaceutical Label Quality Control with Hierarchical RAG and Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces ClinicalRAG, a hierarchical RAG pipeline with LLM query synthesis, DFS table retrieval, and self-consistency TableQA verification to automate pharmaceutical drug label quality control.
Problem
Drug label creation and quality control is labor-intensive and manual; each quantitative statement in a label must be checked against clinical trial tables, and LLMs hallucinate.
Approach
Organizes clinical trial tables into a hierarchical database (compound, indication, protocol, leaf tables), uses an LLM to decompose label paragraphs into statements with hierarchical queries, retrieves tables via depth-first search plus LLM reranking, and verifies statements with self-consistency TableQA, flagging results with color codes in a UI.
Key results
- ClinicalRAG reaches 92% top-1 accuracy (0.84 sensitivity, 1.00 specificity) versus 72% for Vanilla RAG and 59% for the GPT-4 whole-table baseline.
- With top-3 retrieval, ClinicalRAG achieves 95% accuracy, compared with 87% for Vanilla RAG and 65% for GPT-4.
- Reports 96.1% accuracy in internal validation in the abstract.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ClinicalRAG instead of Vanilla RAG | KEYTRUDA label verification · Top-1 | Accuracy: 0.72 -> 0.92 | Helped |
| Use ClinicalRAG instead of GPT4 | KEYTRUDA label verification · Top-1 | Accuracy: 0.59 -> 0.92 | Helped |
| Use ClinicalRAG instead of Vanilla RAG | KEYTRUDA label verification · Top-3 | Accuracy: 0.87 -> 0.95 | Helped |
| Use ClinicalRAG instead of GPT4 | KEYTRUDA label verification · Top-3 | Accuracy: 0.65 -> 0.95 | Helped |
| Use ClinicalRAG instead of Vanilla RAG | KEYTRUDA label verification · Top-1 | Specificity: 0.97 -> 1.00 | Helped |
| Use ClinicalRAG instead of GPT4 | KEYTRUDA label verification · Top-1 | Specificity: 0.33 -> 1.00 | Helped |
Surprising findings
- Feeding all tables into GPT-4's context performed worst, with 0.33 specificity at top-1, indicating many false verifications.
Limitations
- Evaluation is a small internal case study on a single drug label (KEYTRUDA) with 136 statements.
- Synthetic false data generated by GPT-4 rather than real errors.
- Underlying LLM and implementation details are not specified.
Experimental setup
Datasets: KEYTRUDA label document (FDA) internal clinical trial RTF tables converted to markdown
Benchmarks: KEYTRUDA label quantitative statements (68 expert-annotated plus 68 synthetic false)
Metrics: sensitivity specificity accuracy
Baselines: GPT-4 Naive RAG
Builds on: retrieval-augmented generation self-consistency
Closest papers by meaning
- ClinicalLab: Aligning Agents for Multi-Departmental Clinical Diagnostics in the Real World (NeurIPS 2025 · similarity 0.84)
- Deceptive Grounding: Entity Attribution Failure in Clinical Retrieval-Augmented Generation (NeurIPS 2026 · similarity 0.81)
- MedChain: Bridging the Gap Between LLM Agents and Clinical Practice with Interactive Sequence (NeurIPS 2025 · similarity 0.81)
- Entity Only vs. Inline Approaches: Evaluating LLMs for Adverse Drug Event Detection in Clinical Text (Student Abstract) (AAAI 2025 · similarity 0.81)
- Fine-grained List-wise Alignment for Generative Medication Recommendation (NeurIPS 2025 · similarity 0.80)
- MedSafetyBench: Evaluating and Improving the Medical Safety of Large Language Models (NeurIPS 2024 · similarity 0.80)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.