Research map / Large language models / Retrieval and knowledge
Harnessing Large Language Models for Knowledge Graph Question Answering via Adaptive Multi-Aspect Retrieval-Augmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AMAR, which retrieves entity, relation, and subgraph knowledge over Freebase, projects each retrieved text into soft prompt embeddings with self-alignment and relevance gating, and fine-tunes LLaMA2 with LoRA to generate S-expressions for KGQA.
Problem
LLMs hallucinate and lack up-to-date knowledge for KGQA, and appending large volumes of retrieved KG text as context introduces noise that misleads the model's attention to the question.
Approach
Map each retrieved entity, relation, and subgraph text to a prompt embedding, align commonalities across aspects via self- and cross-attention to form consistency tokens, and weight them with a siamese relevance gate before feeding them to a LoRA-tuned LLM that generates S-expressions executed on Freebase.
Key results
- AMAR (LLaMA2-7B) reaches 81.2 F1, 84.3 Hits@1, 75.2 Acc on WebQSP and 78.5 F1, 83.1 Hits@1, 74.5 Acc on CWQ, the best reported among compared methods in the non-oracle setting.
- Ablation on WebQSP: removing all modules (directly appending retrieved text as context) drops F1 from 81.2 to 76.2 and EM from 63.9 to 59.7.
- Under oracle entity linking, AMAR* reaches 84.1 F1 on WebQSP and 82.0 F1 on CWQ.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use AMAR instead of ChatKBQA | KGQA · WebQSP | Acc: 75.2 vs 73.8 | Helped |
| use AMAR instead of ChatKBQA | KGQA · CWQ | F1: 78.5 vs 77.8 | Helped |
| use AMAR instead of ChatKBQA | KGQA · CWQ | Hits@1: 83.1 vs 82.7 | Helped |
| use AMAR instead of ChatKBQA | KGQA · CWQ | Acc: 74.5 vs 73.3 | Helped |
| add the siamese network for relevance scoring (ablation) | AMAR · WebQSP | F1: 80.1 -> 81.2 | Helped |
| add the self-alignment module (ablation) | AMAR · WebQSP | F1: 79.5 -> 81.2 | Helped |
| add the relevance gating module (ablation) | AMAR · WebQSP | F1: 79.4 -> 81.2 | Helped |
| add self-alignment and relevance gating instead of using only MLP prompt embeddings (ablation) | AMAR · WebQSP | F1: 78.7 -> 81.2 | Helped |
| add self-alignment and the siamese network (ablation) | AMAR · WebQSP | F1: 79.1 -> 81.2 | Helped |
| convert retrieved knowledge to prompt embeddings instead of appending it as context (ablation) | AMAR · WebQSP | F1: 76.2 -> 81.2 | Helped |
| add relation retrieval knowledge (ablation) | AMAR · WebQSP | F1: 78.6 -> 81.2 | Helped |
| add entity retrieval knowledge (ablation) | AMAR · WebQSP | F1: 79.2 -> 81.2 | Helped |
Surprising findings
- LLaMA2-13B underperforms LLaMA2-7B on WebQSP, which the authors attribute to overfitting on the smaller, 2-hop-only dataset.
- Removing the relevance-gating siamese network improves BM (77.0 vs 76.4) while lowering EM, suggesting the correct logical form is often ranked at beam positions 2 or lower.
Limitations
- Retrieval relies on a multi-stage pipeline (ELQ, FACC1, BERT relation retriever, BM25 subgraphs) that is complex and dataset-specific.
- Evaluation is limited to two Freebase KGQA datasets (WebQSP and CWQ) with S-expression generation.
Experimental setup
Datasets: WebQSP CWQ Freebase
Benchmarks: WebQSP CWQ
Metrics: F1 Hits@1 Acc EM BM
Baselines: ChatKBQA ToG-R (GPT-4) GNN-RAG G-Retriever DecAF GMT-KBQA UniKGQA RoG
Builds on: ChatKBQA LoRA ELQ SimCSE BM25
Closest papers by meaning
- Simple is Effective: The Roles of Graphs and Large Language Models in Knowledge-Graph-Based Retrieval-Augmented Generation (ICLR 2025 · similarity 0.93)
- Debate on Graph: A Flexible and Reliable Reasoning Framework for Large Language Models (AAAI 2025 · similarity 0.91)
- Think-on-Graph 2.0: Deep and Faithful Large Language Model Reasoning with Knowledge-guided Retrieval Augmented Generation (ICLR 2025 · similarity 0.91)
- Graph Reasoning Transformers for Knowledge-Aware Question Answering (AAAI 2024 · similarity 0.90)
- Knowledge Editing with Dynamic Knowledge Graphs for Multi-Hop Question Answering (AAAI 2025 · similarity 0.90)
- Knowledge Graph Prompting for Multi-Document Question Answering (AAAI 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.