Research map / Large language models / Language, safety and interpretability
Deep Bidirectional Language-Knowledge Graph Pretraining
Michihiro Yasunaga, Antoine Bosselut, Hongyu Ren, Xikun Zhang, Christopher D Manning, Percy Liang, Jure Leskovec
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add joint MLM + LinkPred pretraining instead of MLM only (ablation) | DRAGON · OBQA | accuracy: 67.2 -> 72.0 | Improved |
| add joint MLM + LinkPred pretraining instead of MLM only (ablation) | DRAGON · CSQA | accuracy: 74.3 -> 76.0 | Improved |
| add MLM to LinkPred-only pretraining (ablation) | DRAGON · OBQA | accuracy: 66.4 -> 72.0 | Improved |
| bidirectional text-KG fusion instead of concatenating at end (ablation) | DRAGON · OBQA | accuracy: 68.0 -> 72.0 | Improved |
| use KG as graph instead of converting to sentences (ablation) | DRAGON · OBQA | accuracy: 70.1 -> 72.0 | Improved |
| DistMult head instead of TransE or RotatE (ablation) | DRAGON · OBQA | accuracy: 71.4 -> 72.0 | Improved |
| KG-augmented pretraining vs finetuning-only GreaseLM | DRAGON · OBQA | accuracy: 66.9 -> 72.0 | Improved |
| KG-augmented pretraining on 10% finetuning data | DRAGON · PIQA (10% train) | accuracy: 67.0 -> 72.3 | Improved |
| increase fusion layers from 5 to 7 without pretraining (ablation) | GreaseLM · OBQA | accuracy: 66.9 -> 66.2 | Hurt |
| increase fusion layers from 5 to 7 with pretraining (ablation) | DRAGON · OBQA | accuracy: 72.0 -> 72.8 | Improved |
| KG-augmented pretraining on biomedical LM | BioLinkBERT-Large · MedQA | accuracy: 44.6 -> 47.5 | Improved |
| KG-augmented pretraining on complex reasoning (negation) | RoBERTa · CSQA+OBQA dev negation | accuracy: 61.7 -> 75.2 | Improved |
Closest papers by meaning
- LightPROF: A Lightweight Reasoning Framework for Large Language Model on Knowledge Graph (AAAI 2025 · similarity 0.88)
- Toward a Unified Statistical Theory of Unsupervised Pretraining and Supervised Neural Knowledge Graph Learning (NeurIPS 2026 · similarity 0.87)
- Think-on-Graph: Deep and Responsible Reasoning of Large Language Model on Knowledge Graph (ICLR 2024 · similarity 0.87)
- Harnessing Large Language Models for Knowledge Graph Question Answering via Adaptive Multi-Aspect Retrieval-Augmentation (AAAI 2025 · similarity 0.87)
- Debate on Graph: A Flexible and Reliable Reasoning Framework for Large Language Models (AAAI 2025 · similarity 0.87)
- Reasoning of Large Language Models over Knowledge Graphs with Super-Relations (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
