Research map / Large language models / Retrieval and knowledge
Knowledge Graph Prompting for Multi-Document Question Answering
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Knowledge Graph Prompting (KGP), building a passage/structure KG over multiple documents and using a fine-tuned LLM traversal agent to gather context for multi-document QA.
Problem
Multi-document question answering requires alternately retrieving and reasoning across passages and document structures (pages, tables), which retrieve-and-read methods and prompting LLMs back and forth handle poorly.
Approach
Build a generally applicable KG over documents (passage nodes linked by TF-IDF keyword, semantic KNN, or TAGME entity overlap; page/table structural nodes), then use an instruction-tuned LLM (T5-large, LLaMA-7B) as a graph traversal agent that generates the next evidence and selects the most matching neighbor passage to feed ChatGPT.
Key results
- KGP-T5 achieves the best F1 among non-Golden methods on 2WikiMQA (53.50) and MuSiQue (41.19), vs. MDR at 52.44 and 37.03.
- KGP-T5 reaches 67.00 Struct-EM on the PDFTriage structural questions, where no baseline handles structure queries.
- On HotpotQA, KGP-T5 gets 66.77 F1 and 46.51 EM, outperforming DPR (62.11 F1) and MDR (65.16 F1).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use KGP-T5 instead of the strongest retrieval baseline | MD-QA · HotpotQA | F1: 65.16 -> 66.77 | Helped |
| Use KGP-T5 instead of the strongest retrieval baseline | MD-QA · IIRC | F1: 43.47 -> 41.54 | Hurt |
| Use KGP-T5 instead of the strongest retrieval baseline | MD-QA · 2WikiMQA | F1: 52.44 -> 53.50 | Helped |
| Use KGP-T5 instead of the strongest retrieval baseline | MD-QA · MuSiQue | F1: 37.03 -> 41.19 | Helped |
| Use KGP-T5 to retrieve structural contexts | PDFTriage · PDFTriage | Struct-EM | Helped |
| Use T5 instead of MDR as the KG traversal agent (ablation) | KGP on TAGME-constructed KG · HotpotQA | F1: 65.77 -> 66.77 | Helped |
| Use T5 instead of MDR as the KG traversal agent (ablation) | KGP on TAGME-constructed KG · IIRC | F1: 43.21 -> 41.54 | Hurt |
| Use T5 instead of MDR as the KG traversal agent (ablation) | KGP on TAGME-constructed KG · 2WikiMQA | F1: 51.29 -> 53.50 | Helped |
| Use T5 instead of MDR as the KG traversal agent (ablation) | KGP on TAGME-constructed KG · MuSiQue | F1: 41.11 -> 41.19 | Helped |
| Use an LLM-guided traversal agent instead of random traversal (ablation) | KGP on TAGME-constructed KG · MD-QA | performance | Helped |
| Increase KG density (ablation) | KGP-T5 · HotpotQA | F1 | Helped |
| Increase KG density (ablation) | KGP-T5 · MuSiQue | F1 | Helped |
Surprising findings
- KGP-T5 (T5-large, 0.7B) outperforms LLaMA-7B as the traversal agent, which the authors attribute to LLaMA needing more fine-tuning data.
- Off-the-shelf ChatGPT as a traversal agent with few-shot prompting does not beat the fine-tuned T5 agent on most datasets (e.g., IIRC accuracy 46.27 vs 48.28).
Limitations
- Passage-as-node KG scales poorly for very large document collections; authors estimate it is affordable for only 10-100 documents.
- TAGME graph construction is very slow (over 8 hours for 12 documents), and markdown table input is infeasible for long tables due to ChatGPT token limits.
- Structural questions rely on ChatGPT to extract page/table names; no trained structure extractor is provided.
Experimental setup
Datasets: HotpotQA IIRC 2WikiMQA MuSiQue PDFTriage WikiMHop
Benchmarks: HotpotQA IIRC 2WikiMQA MuSiQue PDFTriage
Metrics: Acc EM F1 Struct-EM SF-EM precision latency
Baselines: None KNN TF-IDF BM25 DPR MDR IRCoT Golden
Builds on: MDR DPR IRCoT TAGME LoRA
Closest papers by meaning
- Harnessing Large Language Models for Knowledge Graph Question Answering via Adaptive Multi-Aspect Retrieval-Augmentation (AAAI 2025 · similarity 0.90)
- Knowledge Editing with Dynamic Knowledge Graphs for Multi-Hop Question Answering (AAAI 2025 · similarity 0.89)
- Simple is Effective: The Roles of Graphs and Large Language Models in Knowledge-Graph-Based Retrieval-Augmented Generation (ICLR 2025 · similarity 0.89)
- Debate on Graph: A Flexible and Reliable Reasoning Framework for Large Language Models (AAAI 2025 · similarity 0.89)
- A Systematic Exploration of Knowledge Graph Alignment with Large Language Models in Retrieval Augmented Generation (AAAI 2025 · similarity 0.88)
- Graph Reasoning Transformers for Knowledge-Aware Question Answering (AAAI 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.