Research map / Graph learning / Knowledge and temporal graphs
EMERGE: A Benchmark for Updating Knowledge Graphs with Emerging Textual Knowledge
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces EMERGE, a benchmark for text-driven knowledge graph updating (TKGU) built from aligned Wikipedia passages and Wikidata snapshot deltas (2019–2025), and benchmarks IE models on five update operations.
Problem
Existing information extraction methods extract triples from text independently of the current KG state, so they cannot determine how new textual knowledge should update a KG (add, mint entities, infer links, deprecate, or detect existing facts).
Approach
Defines text-driven KG updating as a set of edit operations (Exists, Add, Mint+Add, Infer, Deprecate) conditioned on a KG snapshot; builds a dataset by temporally aligning Wikipedia revisions with Wikidata snapshot deltas, filtering with an LLM, and validating with human annotation; extends EDC to EDC+ for Infer and Deprecate.
Key results
- EDC+ with GPT-5.1 achieves the best completeness on all operations, e.g., 25.7 on Exists, 40.2 on Add, 35.6 on Mint+Add, 21.9 on Infer, and 34.8 on Deprecate.
- Completeness is low across all models (max 40.2), while G-BERTScore-R is higher (e.g., 77.5 for EDC+ GPT-5.1 on Add), indicating semantically similar but non-matching predictions.
- LLM annotation agreement with human annotators reaches Cohen's kappa 0.64–0.86 per operation, with overall Fleiss' kappa 0.767 across three annotators.
Surprising findings
- Disabling in-context demonstrations for EDC+ with GPT-5.1 drops completeness by up to 18.9 points, suggesting LLMs exploit implicit KG content from examples.
- Performance drops around the 2021 snapshot for Mint+Add and Infer, attributed to US presidential transition events.
Limitations
- Existing IE models do not canonicalize triples to KG identifiers, so they cannot perform state-dependent updates.
- Ground-truth labels are LLM-generated (Llama-3.1-405B) and only a 500-triple subset is human-validated.
- Completeness metric depends on a cosine threshold of 0.9 and embedding choice, and precision/F1 are not reported due to open-world assumption.
Experimental setup
Datasets: Wikipedia revision dumps Wikidata revision dumps
Benchmarks: EMERGE
Metrics: Completeness G-BERTScore-R
Baselines: REBEL ReLiK RE ReLiK cIE KGGen RAKG EDC+
Builds on: EDC Wikidata5M BERTScore ReLiK REBEL
Closest papers by meaning
- KGGen: Extracting Knowledge Graphs from Plain Text with Language Models (NeurIPS 2025 · similarity 0.87)
- Select and Augment: Enhanced Dense Retrieval Knowledge Graph Augmentation (Abstract Reprint) (AAAI 2024 · similarity 0.86)
- A Systematic Exploration of Knowledge Graph Alignment with Large Language Models in Retrieval Augmented Generation (AAAI 2025 · similarity 0.85)
- Learning Pattern-Based Extractors from Natural Language and Knowledge Graphs: Applying Large Language Models to Wikipedia and Linked Open Data (AAAI 2024 · similarity 0.85)
- WikiBigEdit: Understanding the Limits of Lifelong Knowledge Editing in LLMs (ICML 2025 · similarity 0.85)
- TGB 2.0: A Benchmark for Learning on Temporal Knowledge Graphs and Heterogeneous Graphs (NeurIPS 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.