Research map / Large language models / Retrieval and knowledge
Semantic-KG: Using Knowledge Graphs to Construct Benchmarks for Measuring Semantic Similarity
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds Semantic-KG, a pipeline that perturbs knowledge-graph subgraphs and uses GPT-4o to verbalize them into statement pairs, producing semantic similarity benchmarks across four domains and evaluating NLP metrics, embeddings and LLM judges on them.
Problem
Existing semantic similarity benchmarks are costly to create with human judgment, scarce for specialized domains, and have unclear definitions of semantic equivalence, while current metrics may capture surface form rather than meaning.
Approach
Generate labeled similar/dissimilar statement pairs by sampling KG subgraphs, applying controlled perturbations (node/edge removal or replacement) to create dissimilar pairs, verbalizing with an LLM, and validating by reconstructing the KG from the text; this gives label control without human annotation.
Key results
- Across methods, F1 varies with perturbation type: many methods underperform on edge perturbations, and GPT-4o outperforms traditional NLP methods on edge-replacement (interaction beta=0.184, p=0.031).
- Traditional NLP methods such as Sentence-T5-Base outperform most LLMs on node-removal and node-replacement perturbations (Sentence-T5-Base x node-removal beta=0.260, p=0.002).
- Human spot-checks found 99% of generated statements correct and 75% natural.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use LLM-as-a-judge instead of traditional NLP methods for edge perturbations | Semantic similarity methods · Semantic-KG datasets, edge perturbations | F1 | Mixed |
| Use traditional NLP methods instead of most LLM-as-a-judge methods for node perturbations | Semantic similarity methods · Semantic-KG datasets, node perturbations | F1 | Mixed |
| Apply edge-replacement perturbation rather than other perturbation types | Semantic similarity methods · Semantic-KG benchmark | F1: -0.024 | Hurt |
| Apply node-removal perturbation rather than the reference perturbation type | Semantic similarity methods · Semantic-KG benchmark | F1: 0.124 | Helped |
| Apply node-replacement perturbation rather than the reference perturbation type | Semantic similarity methods · Semantic-KG benchmark | F1: 0.155 | Helped |
| Use GPT-4o on edge-replacement examples rather than the reference method | Semantic similarity evaluation · Semantic-KG benchmark, edge replacement | F1: 0.184 | Helped |
| Use Sentence-T5-Base on node-removal examples rather than the reference method | Semantic similarity evaluation · Semantic-KG benchmark, node removal | F1: 0.260 | Helped |
| Use Sentence-T5-Base on node-replacement examples rather than the reference method | Semantic similarity evaluation · Semantic-KG benchmark, node replacement | F1: 0.216 | Helped |
| Use Sentence-T5-Base on Globi rather than the reference method | Semantic similarity evaluation · Globi | F1: -0.413 | Hurt |
| Use ROUGE-1 on FinDKG rather than the reference method | Semantic similarity evaluation · FinDKG | F1: -0.236 | Hurt |
| Use Gemini-1.5 Pro for domain-specific semantic similarity evaluation | LLM-as-a-judge methods · Oregano | F1 | Helped |
Surprising findings
- Sentence-T5-Base has a marked drop on the Globi dataset, while BERTScore outperforms all methods including LLMs on Globi.
- LLMs are not by default the best choice for semantic similarity; no method is consistently superior.
Limitations
- Depends on availability and quality of underlying knowledge graphs.
- Structural perturbations (node/edge removal or replacement) do not capture real-world semantic variation such as intent, tone or implicature.
- Pipeline success declines for subgraphs larger than 12 triples, and LLM-generated text may differ in distribution from real text.
Experimental setup
Datasets: Codex FinDKG Globi Oregano
Benchmarks: Semantic-KG
Metrics: F1 Clopper-Pearson 95% confidence intervals
Baselines: ROUGE-1 ROUGE-2 ROUGE-L BLEU BERTScore Sentence-T5-Base DeBERTa-Large-MNLI Text-embedding-3-large Text-embedding-ada-002
Builds on: BLEU ROUGE BERTScore
Closest papers by meaning
- A Systematic Exploration of Knowledge Graph Alignment with Large Language Models in Retrieval Augmented Generation (AAAI 2025 · similarity 0.88)
- Simple is Effective: The Roles of Graphs and Large Language Models in Knowledge-Graph-Based Retrieval-Augmented Generation (ICLR 2025 · similarity 0.86)
- KGGen: Extracting Knowledge Graphs from Plain Text with Language Models (NeurIPS 2025 · similarity 0.86)
- MKGL: Mastery of a Three-Word Language (NeurIPS 2024 · similarity 0.86)
- Neither Valid nor Reliable? Investigating the Use of LLMs as Judges (NeurIPS 2025 · similarity 0.86)
- Harnessing Large Language Models for Knowledge Graph Question Answering via Adaptive Multi-Aspect Retrieval-Augmentation (AAAI 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.