Research map / Graph learning / Knowledge and temporal graphs
KnowFormer: Revisiting Transformers for Knowledge Graph Reasoning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes KnowFormer, a text-free transformer for knowledge graph reasoning whose attention uses query-prototype-based kernels computed by relational message-passing networks, with a linear-complexity approximation of the exponential kernel.
Problem
Path-based knowledge graph reasoning methods suffer from missing paths and over-squashing in message passing, while PLM-based transformer methods rely on textual descriptions and fail to capture structure in domain-specific KGs.
Approach
Redefine self-attention over entities as a weighted aggregation based on query prototypes (plausibility of entity pairs for the query relation), with query and value functions built from relational message-passing networks (Q-RMPNN, V-RMPNN) and a first-order Taylor approximation of the exponential kernel that makes attention linear in entities and facts.
Key results
- Transductive FB15k-237: KnowFormer MRR 0.430, H@10 60.8 vs. AdaProp 0.417 MRR and NBFNet 0.415 MRR.
- Transductive YAGO3-10: MRR 0.615, H@1 54.7, H@10 73.4, outperforming the best baseline (AdaProp 0.573 MRR).
- Inductive (12 splits across FB15k-237, WN18RR, NELL-995): achieves the highest MRR on most splits, e.g., NELL-995 v1 MRR 0.827 vs. InGram 0.697.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add the proposed attention mechanism (ablation) | KnowFormer · FB15k-237 | MRR: 0.417 -> 0.430 | Helped |
| add the query function (ablation) | KnowFormer · FB15k-237 | MRR: 0.422 -> 0.430 | Helped |
| add the value function (ablation) | KnowFormer · FB15k-237 | MRR: 0.367 -> 0.430 | Helped |
| use the proposed kernel instead of an RF-based kernel (ablation) | KnowFormer · FB15k-237 | MRR: 0.419 -> 0.430 | Helped |
| replace the approximate kernel with the full-exponential kernel (ablation) | KnowFormer · UMLS | MRR: 0.971 -> 0.974 | Helped |
| use the approximate kernel instead of the full-exponential kernel (ablation) | KnowFormer · UMLS | MRR: 0.974 -> 0.971 | No clear effect |
Surprising findings
- SimKGC, a pretrained-language-model method, performs best on WN18RR transductive (MRR 0.666) but poorly on inductive splits and YAGO3-10.
- Removing the query function causes only a small drop (0.430 to 0.422 MRR on FB15k-237), suggesting the attention input already retains structure from earlier layers.
Limitations
- Evaluated only on semi-inductive settings with invariant relation sets; full-inductive adaptation left to future work.
- Focuses on tail prediction; head and relation prediction are said to transform into the same form without experiments.
- Gains on WN18RR are not best (SimKGC leads), and inductive gaps are smaller than transductive ones.
Experimental setup
Datasets: FB15k-237 WN18RR NELL-995 YAGO3-10 UMLS
Benchmarks: FB15k-237 WN18RR NELL-995 YAGO3-10 UMLS
Metrics: MRR Hits@1 Hits@10
Baselines: TransE DistMult RotatE HousE DRUM CompGCN RNNLogic NBFNet RED-GNN A*Net AdaProp Ultra
Builds on: NBFNet Simple transformer self-attention (Vaswani et al.) SAT (Structure-Aware Transformer) DistMult RA-WL_2 expressivity framework
Closest papers by meaning
- Graph-constrained Reasoning: Faithful Reasoning on Knowledge Graphs with Large Language Models (ICML 2025 · similarity 0.89)
- Towards Graph Foundation Models: Training on Knowledge Graphs Enables Transferability to General Graphs (NeurIPS 2025 · similarity 0.88)
- Talk like a Graph: Encoding Graphs for Large Language Models (ICLR 2024 · similarity 0.88)
- Understanding Transformer Reasoning Capabilities via Graph Algorithms (NeurIPS 2024 · similarity 0.88)
- EPERM: An Evidence Path Enhanced Reasoning Model for Knowledge Graph Question and Answering (AAAI 2025 · similarity 0.88)
- Reasoning on Graphs: Faithful and Interpretable Large Language Model Reasoning (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.