Research map / Large language models / Retrieval and knowledge
Entity Alignment with Noisy Annotations from Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LLM4EA, an iterative framework that uses GPT-3.5/GPT-4 as a zero-shot annotator for entity alignment, with active source-entity selection and a probabilistic label refiner that filters noisy pseudo-labels before training Dual-AMN.
Problem
Entity alignment between knowledge graphs normally needs costly human-labeled seed alignments; using LLM-generated labels is attractive but the annotation space is huge, the query budget is limited, and LLM labels are noisy.
Approach
Combine budget-limited active selection of source entities (scored by relational and neighbor uncertainty with rank aggregation) with an unsupervised label refiner that uses probabilistic reasoning over relation functionality to keep only structurally compatible LLM pseudo-labels, iteratively feeding back the base EA model's confident predictions.
Key results
- With GPT-3.5 as annotator, LLM4EA reaches Hit@1 74.2% on EN-FR-15K and 97.7% on D-Y-15K, outperforming the best baseline Dual-AMN (51.9% and 85.8%) and RDGCN on the same metrics.
- With GPT-4 as annotator, LLM4EA reaches MRR 95.3 on EN-DE-15K and 92.9 on D-W-15K, above Dual-AMN (93.6 and 86.7 MRR on EN-DE and D-W respectively, with D-W at 86.7).
- GPT-3.5 with a 2x query budget matches or exceeds GPT-4 at default budget (e.g., MRR 86.3 vs 86.0 on EN-FR), with roughly 10x lower actual token cost according to the authors.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use LLM4EA with GPT-3.5 rather than the strongest GPT-3.5-label baseline | LLM4EA · EN-FR-15K | MRR: 74.3±0.3 -> 81.0±0.3 | Helped |
| use LLM4EA with GPT-3.5 rather than the strongest GPT-3.5-label baseline | LLM4EA · EN-DE-15K | MRR: 78.9±0.6 -> 92.6±0.3 | Helped |
| use LLM4EA with GPT-3.5 rather than the strongest GPT-3.5-label baseline | LLM4EA · D-W-15K | MRR: 83.2±0.6 -> 90.9±0.2 | Helped |
| use LLM4EA with GPT-3.5 rather than the strongest GPT-3.5-label baseline | LLM4EA · D-Y-15K | MRR: 91.4±0.1 -> 98.3±0.0 | Helped |
| add the label refiner (ablation) | LLM4EA · EN-FR-15K | MRR: 61.9±0.8 -> 81.0±0.3 | Helped |
| add the label refiner (ablation) | LLM4EA · EN-DE-15K | MRR: 82.6±1.3 -> 92.6±0.3 | Helped |
| add the label refiner (ablation) | LLM4EA · D-W-15K | MRR: 52.9±1.2 -> 90.9±0.2 | Helped |
| add the label refiner (ablation) | LLM4EA · D-Y-15K | MRR: 91.5±0.5 -> 98.3±0.0 | Helped |
| add active selection (ablation) | LLM4EA · EN-FR-15K | MRR: 75.4±2.0 -> 81.0±0.3 | Helped |
| add active selection (ablation) | LLM4EA · EN-DE-15K | MRR: 84.6±0.6 -> 92.6±0.3 | Helped |
| add active selection (ablation) | LLM4EA · D-W-15K | MRR: 86.3±0.6 -> 90.9±0.2 | Helped |
| add active selection (ablation) | LLM4EA · D-Y-15K | MRR: 98.1±0.1 -> 98.3±0.0 | Helped |
Surprising findings
- Without the label refiner, the active-selection variant falls below the base Dual-AMN model on several datasets, indicating active selection depends on reliable feedback.
- Label refiner raises the true positive rate of refined labels above 0.9 even when the initial pseudo-label TPR is around 0.5.
Limitations
- Query budget is distributed evenly across active selection iterations rather than adaptively allocated.
- Base EA model is transductive, so new entities or triples require retraining; no direct support for temporal KGs.
Experimental setup
Datasets: OpenEA D-W-15K V2 OpenEA D-Y-15K V2 OpenEA EN-DE-15K V2 OpenEA EN-FR-15K V2
Benchmarks: OpenEA EN-FR-15K OpenEA EN-DE-15K OpenEA D-W-15K OpenEA D-Y-15K
Metrics: Hit@1 Hit@10 MRR precision recall F1-score TPR
Baselines: IMUSE AlignE BootEA GCNAlign RDGCN Dual-AMN Emb-Match Str-Match PARIS
Builds on: Dual-AMN PARIS BootEA
Closest papers by meaning
- HLMEA: Unsupervised Entity Alignment Based on Hybrid Language Models (AAAI 2025 · similarity 0.92)
- GuideNER: Annotation Guidelines Are Better than Examples for In-Context Named Entity Recognition (AAAI 2025 · similarity 0.88)
- NeuSymEA: Neuro-symbolic Entity Alignment via Variational Inference (NeurIPS 2025 · similarity 0.87)
- Lambda: Learning Matchable Prior For Entity Alignment with Unlabeled Dangling Cases (NeurIPS 2024 · similarity 0.87)
- A Systematic Exploration of Knowledge Graph Alignment with Large Language Models in Retrieval Augmented Generation (AAAI 2025 · similarity 0.87)
- MCL-NER: Cross-Lingual Named Entity Recognition via Multi-View Contrastive Learning (AAAI 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.