Research map / Graph learning / Knowledge and temporal graphs
KGGen: Extracting Knowledge Graphs from Plain Text with Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Presents KGGen, an LM-based text-to-knowledge-graph extractor with iterative embedding-plus-LLM entity and edge resolution, and the MINE benchmark for judging extracted KGs.
Problem
Plain-text-to-knowledge-graph extraction produces sparse graphs with near-unique entities and relation types, and no good benchmarks exist to measure extractor fidelity.
Approach
Extract triples with an LLM in two steps (entities, then relations), aggregate across sources, then iteratively cluster S-BERT embeddings (k-means into 128-item clusters, top-k=16 BM25+semantic neighbours) and have the LLM identify duplicates and pick canonical aliases for both entities and edges.
Key results
- On MINE-1, KGGen scored 66.07% on average versus 47.80% for GraphRAG and 29.84% for OpenIE.
- On MINE-2 (WikiQA), KGGen performs comparably to GraphRAG.
- KGGen reuses each relation type about 10 times on average, while GraphRAG reuses each about 2 times regardless of corpus size.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use KGGen instead of GraphRAG | plain-text KG extraction · MINE-1 | information retention (%): 47.80% -> 66.07% | Helped |
| use KGGen instead of OpenIE | plain-text KG extraction · MINE-1 | information retention (%): 29.84% -> 66.07% | Helped |
| use Claude Sonnet 3.5 rather than Gemini 2.0 Flash | KGGen · MINE-1 | information retention (%): 44 -> 73 | Helped |
| use GPT-4o rather than Gemini 2.0 Flash | KGGen · MINE-1 | information retention (%): 44 -> 66 | Helped |
| use KGGen instead of GraphRAG | triple extraction · Validity of extracted triples | valid triples (%): 0/100 (0%) -> 98/100 (98%) | Helped |
| use KGGen instead of OpenIE | triple extraction · Validity of extracted triples | valid triples (%): 55/100 (55%) -> 98/100 (98%) | Helped |
| use KGGen for entity and relationship extraction | SemEval-2010 Task 8 · 100 randomly selected sentences | both target entities captured (%): 96/100 | Helped |
| use KGGen instead of GraphRAG on a 1,000,000-character corpus | plain-text KG extraction · 1,000,000-character corpus | processing time (s): 2,319 -> 551 | Helped |
| apply KGGen's entity and edge resolution as corpus size grows | KGGen · 1,000,000-character corpus | entity count: 4,602 -> 3,573 | Helped |
Surprising findings
- Gemini 2.0 Flash, the backbone used for KGGen, scored lowest (44%) among tested models on MINE-1.
- GraphRAG produced 0% valid KG triples in a manual 100-triple check, despite comparable downstream RAG performance.
Limitations
- Graphs still show over- or under-de-duplication of entities and relations.
- Benchmarks cover corpora up to about 5M tokens, far from web-scale.
- Domain-specific extraction (e.g., medicine, finance) may be limited by general-purpose LLMs.
Experimental setup
Datasets: WikiQA SemEval-2010 Task 8 The Name of the Wind (novel)
Benchmarks: MINE-1 MINE-2 SemEval-2010 Task 8
Metrics: MINE-1 fact retention score (%) MINE-2 answer accuracy triple validity (%) entity de-duplication ratio edges per unique relation
Baselines: OpenIE GraphRAG
Builds on: GraphRAG OpenIE DSPy BM25
Closest papers by meaning
- A Systematic Exploration of Knowledge Graph Alignment with Large Language Models in Retrieval Augmented Generation (AAAI 2025 · similarity 0.90)
- Select and Augment: Enhanced Dense Retrieval Knowledge Graph Augmentation (Abstract Reprint) (AAAI 2024 · similarity 0.89)
- KG-FIT: Knowledge Graph Fine-Tuning Upon Open-World Knowledge (NeurIPS 2024 · similarity 0.89)
- K-ON: Stacking Knowledge on the Head Layer of Large Language Model (AAAI 2025 · similarity 0.88)
- MKGL: Mastery of a Three-Word Language (NeurIPS 2024 · similarity 0.88)
- KGDM: A Diffusion Model to Capture Multiple Relation Semantics for Knowledge Graph Embedding (AAAI 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.