Research map / Large language models / Retrieval and knowledge
Think-on-Graph 2.0: Deep and Faithful Large Language Model Reasoning with Knowledge-guided Retrieval Augmented Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ToG-2, a training-free hybrid RAG framework that alternates KG relation/entity exploration with document-based context retrieval so each source guides the other, evaluated with GPT-3.5-turbo on seven knowledge-intensive datasets.
Problem
Existing RAG methods either retrieve text by semantic similarity or retrieve KG triples, and loosely combined hybrid approaches do not let one knowledge source improve retrieval on the other, which hurts depth and completeness for complex multi-hop reasoning.
Approach
Tightly couple KG and text retrieval: KG relations and neighbor entities guide document retrieval (knowledge-guided context retrieval), and retrieved entity documents score and prune candidate entities for the next graph-search step (context-enhanced graph search), iterated with LLM sufficiency checks and clue generation.
Key results
- With GPT-3.5-turbo, ToG-2 reaches 81.1% EM on WebQSP vs 76.2% for ToG, and 42.9% EM on AdvHotpotQA vs 26.3% for ToG.
- On the new ToG-FinQA dataset, ToG-2 reaches 34.0% vs 14.0% for ToG, 6.2% for GraphRAG, and 0% for vanilla RAG and CoT.
- ToG-2 lifts Llama-3-8B on AdvHotpotQA from 20.8 (direct) to 34.7 EM, and Qwen2-7B from 17.9 to 30.8.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ToG-2 instead of ToG | GPT-3.5-turbo · AdvHotpotQA | EM: 26.3% -> 42.9% | Helped |
| Use ToG-2 instead of ToG | GPT-3.5-turbo · WebQSP | EM: 76.2% -> 81.1% | Helped |
| Use ToG-2 instead of ToG | GPT-3.5-turbo · QALD-10-en | EM: 50.2% -> 54.1% | Helped |
| Use ToG-2 instead of ToG | GPT-3.5-turbo · FEVER | Accuracy: 52.7% -> 63.1† (59.7%‡); 63.5%† (58.5%‡) | Mixed |
| Use ToG-2 instead of ToG | GPT-3.5-turbo · Creak | Accuracy: 93.8% -> 93.5% | No clear effect |
| Use ToG-2 instead of ToG | GPT-3.5-turbo · Zero-Shot RE | EM: 88.0% -> 91.0% | Helped |
| Use ToG-2 instead of ToG | GPT-3.5-turbo · ToG-FinQA | Accuracy: 14.0% -> 34.0% | Helped |
| Add Topic Prune (ablation) | ToG-2 · WebQSP | EM: 77.6% -> 81.1% | Helped |
| Add retrieval feedback (ablation) | ToG-2 · WebQSP | EM: 78.0% -> 81.1% | Helped |
| Add retrieval feedback (ablation) | ToG-2 · AdvHotpotQA | EM: 40.2% -> 42.8% | Helped |
| Raise relation-pruning threshold from 0.2 to 0.5 (ablation) | ToG-2 · HotpotQA | Results: 42.9 -> 43.2 | No clear effect |
| Raise relation-pruning threshold to 0.8 (ablation) | ToG-2 · HotpotQA | Results: 43.2 -> 40.0 | Hurt |
Surprising findings
- Vanilla RAG and CoT score 0% on ToG-FinQA, indicating that unseen domain-specific knowledge cannot be recovered without deep graph-guided retrieval.
- On Creak, ToG-2 is not significantly better than ToG (p > 0.9) and the authors attribute this to the simplicity of questions and overcautious behavior.
Limitations
- Relies on EM for evaluation, which the authors acknowledge produces false negatives for aliases and differing detail levels.
- Performance degrades substantially when the KG is incomplete (23% EM at 30% completeness with default settings).
- ToG-FinQA is small (97 questions) and manually designed.
Experimental setup
Datasets: WebQSP AdvHotpotQA QALD-10-en FEVER Creak Zero-Shot RE Wikipedia Wikidata
Benchmarks: WebQSP AdvHotpotQA QALD-10-en FEVER Creak Zero-Shot RE ToG-FinQA
Metrics: Exact Match Accuracy
Baselines: Direct reasoning Chain-of-Thought Self-Consistency Vanilla RAG Think-on-Graph Chain-of-Knowledge GraphRAG
Builds on: Think-on-Graph Chain-of-Knowledge Chain-of-Thought ITER-RETGEN
Closest papers by meaning
- Simple is Effective: The Roles of Graphs and Large Language Models in Knowledge-Graph-Based Retrieval-Augmented Generation (ICLR 2025 · similarity 0.95)
- Fast Think-on-Graph: Wider, Deeper and Faster Reasoning of Large Language Model on Knowledge Graph (AAAI 2025 · similarity 0.92)
- A Systematic Exploration of Knowledge Graph Alignment with Large Language Models in Retrieval Augmented Generation (AAAI 2025 · similarity 0.92)
- Harnessing Large Language Models for Knowledge Graph Question Answering via Adaptive Multi-Aspect Retrieval-Augmentation (AAAI 2025 · similarity 0.91)
- Think-on-Graph: Deep and Responsible Reasoning of Large Language Model on Knowledge Graph (ICLR 2024 · similarity 0.91)
- Can Knowledge-Graph-based Retrieval Augmented Generation Really Retrieve What You Need? (NeurIPS 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.