Research map / Large language models / Retrieval and knowledge
Large Language Models-guided Dynamic Adaptation for Temporal Knowledge Graph Reasoning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LLM-DA, which uses ChatGPT to generate and iteratively update temporal logical rules for TKG reasoning, combined with RE-GCN/TiRGN graph scores, without fine-tuning the LLM.
Problem
Temporal knowledge graph reasoning (TKGR) with deep models lacks interpretability and rule-based methods struggle to learn temporal rules; fine-tuning LLMs to track evolving TKG knowledge is costly, and LLM predictions are opaque.
Approach
LLM generates general temporal logical rules from sampled historical paths, filtered by a Sentence-BERT contextual relation selector; a dynamic adaptation step prompts the LLM to rewrite low-confidence rules using rules extracted from current data, and the rules are combined with a graph-based TKGR score, with no LLM fine-tuning.
Key results
- LLM-DA (TiRGN) reaches MRR 0.471 on ICEWS14 and 0.521 on ICEWS05-15, versus 0.441 and 0.495 for TiRGN alone and 0.497 for Mixtral-8x7B-CoH on ICEWS05-15.
- On ICEWS18, LLM-DA (TiRGN) achieves MRR 0.357 and Hit@10 0.570, versus 0.336 and 0.542 for TiRGN.
- Ablation: removing dynamic adaptation drops LLM-DA (TiRGN) MRR on ICEWS14 from 0.471 to 0.457 (H+C variant).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply LLM-DA dynamic adaptation to TiRGN-based reasoning | TiRGN-based TKGR · ICEWS14 | MRR: 0.441 -> 0.471 | Helped |
| apply LLM-DA dynamic adaptation to TiRGN-based reasoning | TiRGN-based TKGR · ICEWS05-15 | MRR: 0.495 -> 0.521 | Helped |
| apply LLM-DA dynamic adaptation to TiRGN-based reasoning | TiRGN-based TKGR · ICEWS18 | MRR: 0.336 -> 0.357 | Helped |
| add current-data rules to historical-data rules (ablation) | LLM-DA (TiRGN) without dynamic adaptation · ICEWS14 | MRR: 0.450 -> 0.457 | Helped |
| add historical-data rules to current-data rules (ablation) | LLM-DA (TiRGN) without dynamic adaptation · ICEWS14 | MRR: 0.454 -> 0.457 | Helped |
| add current-data rules to historical-data rules (ablation) | LLM-DA (TiRGN) without dynamic adaptation · ICEWS05-15 | MRR: 0.503 -> 0.511 | Helped |
| add historical-data rules to current-data rules (ablation) | LLM-DA (TiRGN) without dynamic adaptation · ICEWS05-15 | MRR: 0.508 -> 0.511 | Helped |
| add dynamic adaptation of historical-data rules using current data (ablation) | LLM-DA (TiRGN) with historical and current data · ICEWS14 | MRR: 0.457 -> 0.471 | Helped |
| add dynamic adaptation of historical-data rules using current data (ablation) | LLM-DA (TiRGN) with historical and current data · ICEWS05-15 | MRR: 0.511 -> 0.521 | Helped |
| increase the number of dynamic-adaptation iterations (ablation) | LLM-DA · ICEWS14 and ICEWS05-15 | MRR | Helped |
| set the rule-based reasoning weight to 1, using only rules (ablation) | LLM-DA · ICEWS14 and ICEWS05-15 | MRR | Hurt |
| vary the rule-based reasoning weight α (ablation) | LLM-DA · ICEWS14 and ICEWS05-15 | MRR | Mixed |
Surprising findings
- Rules generated by the LLM alone are often too broad and can underperform traditional TKGR methods such as PPT and GPT-NeoX.
- Setting alpha=1 (rules only, no graph score) lowers MRR, indicating rules alone cannot cover all future-query answers due to distribution shift.
Limitations
- Does not model node semantics, which may reduce rule quality.
- Rules are not generated conditioned on the query, so they can lack specificity.
- Requires manually designed prompts that must be redesigned per dataset, at significant cost.
Experimental setup
Datasets: ICEWS14 ICEWS05-15 ICEWS18
Benchmarks: ICEWS14 ICEWS05-15 ICEWS18
Metrics: MRR Hit@1 Hit@3 Hit@10
Baselines: RE-NET RE-GCN TiRGN TLogic PPT Llama-2-7b-CoH Vicuna-7b-CoH GPT-NeoX Mixtral-8x7B-CoH
Builds on: TLogic TiRGN RE-GCN Sentence-BERT
Closest papers by meaning
- Integrating Large Language Models and Möbius Group Transformations for Temporal Knowledge Graph Embedding on the Riemann Sphere (AAAI 2025 · similarity 0.90)
- LLM-DR: A Novel LLM-Aided Diffusion Model for Rule Generation on Temporal Knowledge Graphs (AAAI 2025 · similarity 0.90)
- Think-on-Graph: Deep and Responsible Reasoning of Large Language Model on Knowledge Graph (ICLR 2024 · similarity 0.89)
- Reasoning on Graphs: Faithful and Interpretable Large Language Model Reasoning (ICLR 2024 · similarity 0.88)
- TGB 2.0: A Benchmark for Learning on Temporal Knowledge Graphs and Heterogeneous Graphs (NeurIPS 2024 · similarity 0.87)
- TEILP: Time Prediction over Knowledge Graphs via Logical Reasoning (AAAI 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.