Research map / Large language models / Retrieval and knowledge
Debate on Graph: A Flexible and Reliable Reasoning Framework for Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DoG, an iterative KGQA framework where LLMs interact with knowledge graphs, try answering after each reasoning step, and use a three-role debate team to simplify multi-hop questions.
Problem
LLM-based knowledge graph question answering suffers from excessively long evidence paths that distract answer generation and from false-positive relations that hinder path refinement.
Approach
Combines subgraph-focusing answer trying after each hop with a multi-role LLM debate team (expert, critic, linguist) that iteratively rewrites a K-hop question into a (K-1)-hop question based on the retrieved triple, so already-resolved relations no longer interfere; all done via in-context learning without training.
Key results
- DoG with GPT-3.5-Turbo outperforms ToG by 23.7% on WebQuestions and 9.1% on GrailQA in accuracy.
- DoG with GPT-3.5-Turbo reaches 88.6 on WebQSP, exceeding the best supervised UniKGQA (77.2) by 11.4 points.
- Removing the debate team and using a single agent for all three roles decreases average performance by 10.3% relative to the full model.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DoG with GPT-3.5-Turbo instead of ToG | KGQA · WebQuestions | accuracy: 54.5 -> 78.2 | Helped |
| Use DoG with GPT-3.5-Turbo instead of ToG | KGQA · GrailQA | accuracy: 68.7 -> 77.8 | Helped |
| Use DoG with GPT-3.5-Turbo instead of UniKGQA | KGQA · WebQSP | accuracy: 77.2 -> 88.6 | Helped |
| Remove subgraph focusing and question simplifying (ablation) | DoG with GPT-3.5-Turbo · MetaQA 2-hop | accuracy: 96.6 -> 76.6 | Hurt |
| Remove subgraph focusing and question simplifying (ablation) | DoG with GPT-3.5-Turbo · MetaQA 3-hop | accuracy: 90.9 -> 38.8 | Hurt |
| Add subgraph focusing and question-simplifying expert R1 (ablation) | DoG with GPT-3.5-Turbo · MetaQA 2-hop | accuracy: +14.8 | Helped |
| Add critic R2 to SF and R1 (ablation) | DoG with GPT-3.5-Turbo · WebQSP | accuracy: +6.2 | Helped |
| Add linguist R3 to SF, R1 and R2 (ablation) | DoG with GPT-3.5-Turbo · CWQ | accuracy: +15.2 | Helped |
| Fuse the three question-simplifying roles into one agent instead of debating (ablation) | DoG with GPT-3.5-Turbo · five-dataset average | average performance increase: 10.3% | Hurt |
| Increase the number of debate rounds (ablation) | DoG with GPT-3.5-Turbo · majority of datasets | accuracy | Hurt |
| Increase the number of relation-filtering exemplars (ablation) | DoG · majority of datasets | accuracy | Helped |
| Increase the number of question-simplifying exemplars (ablation) | DoG · five public datasets | accuracy | No clear effect |
Surprising findings
- Increasing the number of debate rounds often decreases performance, as agents get influenced by others' views in the one-by-one discussion.
Limitations
- Relation filtering errors are common when many relations link to an entity.
- Evaluation uses sampled subsets (500 per dataset, 100 for GPT-4), and fallback to parametric knowledge when iteration limit is reached.
- Performance with Qwen-14B and Llama-3-8B is lower than GPT-3.5 on some datasets (e.g., Qwen on CWQ and MetaQA 2/3-hop).
Experimental setup
Datasets: MetaQA WebQSP CWQ WebQuestions GrailQA
Benchmarks: MetaQA WebQSP CWQ WebQuestions GrailQA
Metrics: exact match accuracy (Hits@1)
Baselines: KV-Mem GraftNet PullNet EmbedKGQA NSM TransferNet UniKGQA StructGPT KG-GPT KB-BINDER ToG
Builds on: ToG StructGPT chain-of-thought
Closest papers by meaning
- EPERM: An Evidence Path Enhanced Reasoning Model for Knowledge Graph Question and Answering (AAAI 2025 · similarity 0.92)
- Reasoning on Graphs: Faithful and Interpretable Large Language Model Reasoning (ICLR 2024 · similarity 0.92)
- Harnessing Large Language Models for Knowledge Graph Question Answering via Adaptive Multi-Aspect Retrieval-Augmentation (AAAI 2025 · similarity 0.91)
- Graph-constrained Reasoning: Faithful Reasoning on Knowledge Graphs with Large Language Models (ICML 2025 · similarity 0.91)
- Think-on-Graph: Deep and Responsible Reasoning of Large Language Model on Knowledge Graph (ICLR 2024 · similarity 0.91)
- Reasoning of Large Language Models over Knowledge Graphs with Super-Relations (ICLR 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.