Research map / Large language models / Retrieval and knowledge
Simple is Effective: The Roles of Graphs and Large Language Models in Knowledge-Graph-Based Retrieval-Augmented Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SubgraphRAG, a KG-based RAG framework that retrieves triples in parallel with a lightweight MLP scorer using directional distance encoding, then prompts unfine-tuned LLMs to answer over the retrieved subgraph.
Problem
KG-based RAG must balance retrieval effectiveness and efficiency, selecting a suitable amount of relevant graph information that fits an LLM's reasoning capacity, for multi-hop KGQA.
Approach
Factorize the subgraph retriever over triples and score each triple in parallel with an MLP over text embeddings plus directional distance encodings of entities relative to topic entities; the top-K triples form a flexible-size subgraph fed to a frozen LLM with ICL prompts.
Key results
- SubgraphRAG retriever reaches 0.883 shortest-path triple recall on WebQSP and 0.811 on CWQ, versus 0.714 for cosine similarity and 0.623 for RoG on CWQ, while running in about 6 s on WebQSP.
- With GPT-4o and 100 triples, SubgraphRAG achieves 76.46 Macro-F1 and 89.80 Hit on WebQSP and 59.08 Macro-F1 and 66.69 Hit on CWQ; Llama3.1-8B reaches 70.57 Macro-F1 on WebQSP.
- On WebQSP-sub with GPT-4o, SubgraphRAG reaches 77.61 Macro-F1 and 91.40 Hit, and with 200 triples GPT-4o gets 66.48 Macro-F1 on CWQ-sub.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use SubgraphRAG retrieval instead of RoG retrieval | KG subgraph retrieval · WebQSP | GPT-4o triple recall: 0.388 -> 0.865 | Helped |
| use SubgraphRAG retrieval instead of GNN-RAG | KG subgraph retrieval · CWQ | answer entity recall: 0.841 -> 0.914 | Helped |
| use SubgraphRAG retrieval instead of GNN-RAG | KG subgraph retrieval · WebQSP | retrieval time (s): 68 -> 6 | Helped |
| add directional distance encoding (DDE) to the MLP retriever (ablation) | SubgraphRAG retriever · CWQ, ≥3-hop questions | GPT-4o triple recall: 0.616 -> 0.755 | Helped |
| add a one-hot topic entity indicator (ablation) | MLP retriever · CWQ, ≥3-hop questions | GPT-4o triple recall: 0.616 -> 0.647 | Helped |
| replace MLP with a GNN retriever (ablation) | SubgraphRAG retrieval design · CWQ retrieval | recall | Hurt |
| add Personalized PageRank to the topic-entity MLP (ablation) | SubgraphRAG retrieval design · CWQ retrieval | recall | No clear effect |
| retrieve more triples (ablation) | SubgraphRAG + Llama3.1-8B · WebQSP and CWQ | question-answering performance | Hurt |
| retrieve more triples (ablation) | SubgraphRAG + GPT-4o-mini · WebQSP and CWQ | question-answering performance | Helped |
| replace SubgraphRAG retrieval with random triple sampling (ablation) | SubgraphRAG + Llama3.1-8B-Instruct · CWQ-sub | Macro-F1: 54.76 -> 29.97 | Hurt |
| remove KG retrieval (ablation) | SubgraphRAG + Llama3.1-8B-Instruct · CWQ-sub | Macro-F1: 54.76 -> 27.42 | Hurt |
| replace SubgraphRAG retrieval with GNN-RAG (ablation) | SubgraphRAG + GPT4o-mini · CWQ-sub | Macro-F1: 61.13 -> 60.89 | No clear effect |
Surprising findings
- Larger retrieval budgets hurt Llama3.1-8B markedly while GPT-4o-mini improves with more triples, showing LLM-dependent optimal subgraph size.
- RoG's recall drops sharply when evaluated on GPT-4o-labeled relevant triples instead of shortest-path triples, suggesting path-constrained retrieval misses flexible evidence.
Limitations
- Relies on shortest paths between topic and answer entities as weak supervision, which may miss other relevant triples.
- Smaller LLMs such as Llama3.1-8B degrade with larger retrieved subgraphs, so subgraph size must be tuned per LLM.
- Possible label leakage between WebQSP and CWQ training and test sets affects cross-dataset comparisons.
Experimental setup
Datasets: WebQSP CWQ Freebase
Benchmarks: WebQSP CWQ WebQSP-sub CWQ-sub
Metrics: Macro-F1 Micro-F1 Hit Hit@1 Score_h triple recall answer entity recall wall-clock time
Baselines: cosine similarity SR+NSM w E2E Retrieve-Rewrite-Answer RoG G-Retriever GNN-RAG UniKGQA KD-CoT ToG StructGPT EtD
Builds on: G-Retriever RoG GNN-RAG distance encoding labeling trick GraphSAGE
Closest papers by meaning
- Think-on-Graph 2.0: Deep and Faithful Large Language Model Reasoning with Knowledge-guided Retrieval Augmented Generation (ICLR 2025 · similarity 0.95)
- A Systematic Exploration of Knowledge Graph Alignment with Large Language Models in Retrieval Augmented Generation (AAAI 2025 · similarity 0.94)
- Harnessing Large Language Models for Knowledge Graph Question Answering via Adaptive Multi-Aspect Retrieval-Augmentation (AAAI 2025 · similarity 0.93)
- GFM-RAG: Graph Foundation Model for Retrieval Augmented Generation (NeurIPS 2025 · similarity 0.92)
- Can Knowledge-Graph-based Retrieval Augmented Generation Really Retrieve What You Need? (NeurIPS 2025 · similarity 0.92)
- Fast Think-on-Graph: Wider, Deeper and Faster Reasoning of Large Language Model on Knowledge Graph (AAAI 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.