Research map / Multi-agent systems and games / LLM agents
KARMA: Leveraging Multi-Agent LLMs for Automated Knowledge Graph Enrichment
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes KARMA, a nine-agent LLM pipeline that parses PubMed articles and integrates extracted triplets into existing biomedical knowledge graphs, evaluated with LLM-based and graph-level metrics across three domains.
Problem
Manual curation cannot keep knowledge graphs up to date with the growth of scientific literature; automated LLM extraction suffers from hallucination, schema inconsistency and conflicts with existing KG facts.
Approach
A hierarchical multi-agent LLM pipeline (ingestion, reader, summarizer, entity, relation, schema alignment, conflict resolution, evaluator agents) where cross-agent verification, LLM debate for conflicts, and weighted confidence/clarity/relevance scoring filter triplets before KG integration.
Key results
- DeepSeek-v3 backbone yields 38,230 new entities in genomics (coverage gain) with 0.831 LLM-based correctness.
- Ablation: removing the Summarizer Agent raises new-entity count but lowers genomics QA coherence from 0.612 to 0.472.
- Single-agent GLM-4 baseline reaches 4,384 coverage gain in genomics versus 38,230 for KARMA with DeepSeek-v3 (no gold-standard accuracy reported).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add the Summarizer Agent (ablation) | KARMA · Genomics | QA coherence: 0.472 -> 0.612 | Helped |
| Add the Summarizer Agent (ablation) | KARMA · Proteomics | QA coherence: 0.547 -> 0.613 | Helped |
| Add the Summarizer Agent (ablation) | KARMA · Metabolomics | QA coherence: 0.537 -> 0.493 | Hurt |
| Add the Conflict Resolution Agent (ablation) | KARMA · Genomics | QA coherence: 0.554 -> 0.612 | Helped |
| Add the Conflict Resolution Agent (ablation) | KARMA · Proteomics | QA coherence: 0.583 -> 0.613 | Helped |
| Add the Conflict Resolution Agent (ablation) | KARMA · Metabolomics | QA coherence: 0.471 -> 0.493 | Helped |
| Add the Evaluator Agent (ablation) | KARMA · Genomics | QA coherence: 0.561 -> 0.612 | Helped |
| Add the Evaluator Agent (ablation) | KARMA · Proteomics | QA coherence: 0.605 -> 0.613 | Helped |
| Add the Evaluator Agent (ablation) | KARMA · Metabolomics | QA coherence: 0.480 -> 0.493 | Helped |
| Use KARMA instead of Single-Agent | Genomics KG enrichment · Genomics | LLM-based correctness: 0.493 -> 0.831 | Helped |
| Use KARMA instead of Single-Agent | Proteomics KG enrichment · Proteomics | LLM-based correctness: 0.638 -> 0.772 | Helped |
| Use KARMA instead of Single-Agent | Metabolomics KG enrichment · Metabolomics | LLM-based correctness: 0.527 -> 0.668 | Helped |
Surprising findings
- Removing the Summarizer Agent increases entity and triple yield but reduces accuracy, indicating noise trade-off.
Limitations
- Evaluation relies mainly on LLM-based judgments rather than gold-standard or expert validation.
- Performance varies across domains; metabolomics has lower QA coherence due to sparse relations.
- No gold-standard KG reference; correctness is judged by an LLM (DeepSeek-v3) that is also the backbone.
Experimental setup
Datasets: PubMed articles (genomics, proteomics, metabolomics corpora)
Benchmarks: PubMed corpora (genomics, proteomics, metabolomics)
Metrics: average confidence average clarity average relevance coverage gain connectivity gain conflict ratio LLM-based correctness QA coherence human evaluation score
Baselines: Single-Agent (GLM-4)
Builds on: AutoGen
Closest papers by meaning
- Think-on-Graph: Deep and Responsible Reasoning of Large Language Model on Knowledge Graph (ICLR 2024 · similarity 0.87)
- Graph2Eval: Automatic Multimodal Task Generation for Agents via Knowledge Graphs (CVPR 2026 · similarity 0.87)
- A Systematic Exploration of Knowledge Graph Alignment with Large Language Models in Retrieval Augmented Generation (AAAI 2025 · similarity 0.86)
- KGGen: Extracting Knowledge Graphs from Plain Text with Language Models (NeurIPS 2025 · similarity 0.86)
- GraphMaster: Automated Graph Synthesis via LLM Agents in Data-Limited Environments (NeurIPS 2025 · similarity 0.85)
- Knowledge Tagging with Large Language Model Based Multi-Agent System (AAAI 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.