Research map / Large language models / Retrieval and knowledge
KGARevion: An AI Agent for Knowledge-Intensive Biomedical QA
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces KGARevion, a KG-grounded LLM agent that generates candidate triplets with an LLM, verifies them with a LoRA-fine-tuned reviewer using KG embeddings, revises false ones, and answers from verified triplets, evaluated on medical QA benchmarks and three new MedDDx datasets.
Problem
Biomedical QA with LLMs suffers from hallucinated or incomplete knowledge, and RAG pipelines lack post-retrieval verification, especially for questions needing multiple interrelated concepts or semantically similar answer candidates.
Approach
Generate triplets from the LLM's latent knowledge, then verify each triplet with an LLM fine-tuned on KG completion (with TransE-derived structural embeddings aligned to relation descriptions as prefix tokens), revise triplets judged false, and answer only from verified triplets.
Key results
- KGARevion with LLaMA3.1-8B improves over the best baseline by +5.3% on MMLU-Med and +5.7% on MedQA-US in multiple-choice reasoning.
- On MedDDx-Expert, KGARevion improves accuracy by 3.2% (LLaMA3-8B) and 5.6% (LLaMA3.1-8B) over the best baseline.
- On AfriMed-QA expert multiple-choice, KGARevion improves accuracy by 5.2% with LLaMA3.1-8B and 4.6% with GPT-4-Turbo (zero-shot).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add Review verification to KGARevion (ablation) | KGARevion with LLaMA3-8B · medical QA datasets, multiple-choice | average accuracy: 9% | Helped |
| add Review verification to KGARevion (ablation) | KGARevion with LLaMA3-8B · medical QA datasets, open-ended | average accuracy: 4% | Helped |
| add iterative Revise refinement (ablation) | KGARevion · MedDDx, multiple-choice | accuracy: 3.3% | Helped |
| add iterative Revise refinement (ablation) | KGARevion · MedDDx, open-ended | accuracy: 3.0% | Helped |
| increase revision rounds from k=1 to k=2 or k=3 (ablation) | KGARevion · multiple-choice medical QA | accuracy | Mixed |
| use KGARevion instead of the strongest baseline | medical QA benchmarks · MMLU-Med | accuracy: +5.3% | Helped |
| use KGARevion instead of the strongest baseline | medical QA benchmarks · MedQA-US | accuracy: +5.7% | Helped |
| use KGARevion instead of the strongest baseline | medical QA benchmarks · PubMedQA* | accuracy: +3.8% | Helped |
| use KGARevion instead of the strongest baseline | medical QA benchmarks · BioASQ-Y/N | accuracy: +4.4% | Helped |
| use KGARevion instead of the strongest baseline | medical QA benchmarks · MedDDx-Expert | accuracy: +5.6% | Helped |
| use KGARevion instead of LLM-only backbone | LLaMA3-8B · all datasets | accuracy: 6% | Helped |
| use KGARevion instead of LLM-only backbone | LLaMA3.1-8B · all datasets | accuracy: 7% | Helped |
Surprising findings
- Pure LLM accuracy shifts by 8.4% (LLaMA3-8B) and 16.0% (LLaMA3.1-8B) when answer order changes, while KGARevion is more stable.
- Improvement with GPT-4-Turbo backbone is only about 2%, smaller than for open-source 8B backbones.
Limitations
- Review relies on KG completion fine-tuning; triplets whose entities cannot be mapped to the KG are kept as potentially incomplete knowledge without verification.
- Baseline KG-based models were retrained on MedMCQA rather than their original setups, and open-ended baselines were adapted with extra modules, which may affect comparability.
- Results rely on a single PrimeKG-derived fine-tuning and a modest number of runs (three seeds).
Experimental setup
Datasets: MMLU-Med MedQA-US PubMedQA* BioASQ-Y/N PrimeKG OGB-biokg STaRK-Prime MedMCQA AfriMed-QA
Benchmarks: MMLU-Med MedQA-US PubMedQA* BioASQ-Y/N MedDDx-Basic MedDDx-Intermediate MedDDx-Expert AfriMed-QA
Metrics: accuracy standard deviation over three runs
Baselines: LLaMA2-7B LLaMA3-8B LLaMA3.1-8B Mistral-7B MedAlpaca-7B PMC-LLaMA-7B Llama3-OpenBioLLM-8B Self-RAG MedRAG KG-RAG KG-Rank QAGNN
Builds on: TransE Self-RAG KG-Rank JointLK QAGNN
Closest papers by meaning
- Reasoning-Enhanced Healthcare Predictions with Knowledge Graph Community Retrieval (ICLR 2025 · similarity 0.87)
- MedLA: A Logic-Driven Multi-Agent Framework for Complex Medical Reasoning with Large Language Models (AAAI 2026 · similarity 0.87)
- Visual and Domain Knowledge for Professional-level Graph-of-Thought Medical Reasoning (ICML 2025 · similarity 0.86)
- MDAgents: An Adaptive Collaboration of LLMs for Medical Decision-Making (NeurIPS 2024 · similarity 0.86)
- Simulating Viva Voce Examinations to Evaluate Clinical Reasoning in Large Language Models (NeurIPS 2025 · similarity 0.86)
- MedChain: Bridging the Gap Between LLM Agents and Clinical Practice with Interactive Sequence (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.