Research map / Large language models / Language, safety and interpretability
End-to-End Ontology Learning with Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes OLLM, which finetunes Mistral 7B with a frequency-based masked loss to generate taxonomic subgraphs end-to-end for building ontologies, plus new graph similarity metrics, evaluated on Wikipedia categories and arXiv.
Problem
Ontology learning is usually decomposed into isolated subtasks (concept discovery, relation extraction), which ignores their interactions and does not measure final ontology quality; existing ontology comparison metrics rely on literal text matching.
Approach
Model entire relevant subgraphs of the target ontology as linearised root-to-concept paths with an LLM, trained with a regulariser that randomly masks loss on frequent relations so the model does not overfit high-level concepts; sum generated subgraphs and prune to get the ontology; evaluate with embedding-based Fuzzy, Continuous and Graph F1 plus a motif distance.
Key results
- On Wikipedia, OLLM achieves the best Fuzzy F1 (0.915), Continuous F1 (0.500) and Graph F1 (0.644) among all methods, while Memorisation scores highest on Literal F1 (0.134 vs 0.093 for OLLM).
- On arXiv transfer, OLLM (transfer) reaches Fuzzy F1 0.570, Continuous F1 0.357 and Graph F1 0.633, versus next-best 0.460, 0.290 and 0.546.
- Wikipedia generation took 12 A100-hours for training and 7 A100-hours for inference.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use end-to-end subgraph modeling with masked-loss fine-tuning instead of the strongest baseline | OLLM · Wikipedia | Fuzzy F1: 0.888 -> 0.915 | Helped |
| Use end-to-end subgraph modeling with masked-loss fine-tuning instead of the strongest baseline | OLLM · Wikipedia | Continuous F1: 0.477 -> 0.500 | Helped |
| Use end-to-end subgraph modeling with masked-loss fine-tuning instead of the strongest baseline | OLLM · Wikipedia | Graph F1: 0.639 -> 0.644 | Helped |
| Use end-to-end subgraph modeling with masked-loss fine-tuning instead of the strongest baseline | OLLM · arXiv | Fuzzy F1: 0.460 -> 0.570 | Helped |
| Use end-to-end subgraph modeling with masked-loss fine-tuning instead of the strongest baseline | OLLM · arXiv | Continuous F1: 0.290 -> 0.357 | Helped |
| Use end-to-end subgraph modeling with masked-loss fine-tuning instead of the strongest baseline | OLLM · arXiv | Graph F1: 0.553 -> 0.633 | Helped |
| Add chain-of-thought prompting to Zero-shot (ablation) | Zero-shot prompting · Wikipedia | Continuous F1: 0.455 -> 0.449 | No clear effect |
| Add chain-of-thought prompting to Zero-shot (ablation) | Zero-shot prompting · Wikipedia | Fuzzy F1: 0.871 -> 0.873 | No clear effect |
| Replace end-to-end subgraph modeling with concept discovery and fine-tuned BERT link prediction (ablation) | Zero-shot end-to-end modeling · Wikipedia | Fuzzy F1: 0.871 -> 0.841 | Hurt |
| Replace end-to-end subgraph modeling with concept discovery and fine-tuned BERT link prediction (ablation) | Zero-shot end-to-end modeling · Wikipedia | Motif Distance: 0.341 -> 0.092 | Helped |
Surprising findings
- Zero-shot prompting of Mistral 7B Instruct already outperforms the Hearst and REBEL extraction baselines (even when those are given ground-truth concepts) on the semantic metrics, though prompting scores poorly on motif distance.
- Memorisation scores best on Literal F1 and Finetune/Memorisation achieve the best motif distance, showing syntactic and structural metrics can reward overfitting.
Limitations
- Only concepts and taxonomic relations are modelled; non-taxonomic relations would need new linearisation and metrics.
- Generated taxonomic relations may contain cycles, so transitivity is not guaranteed (97 cycles on Wikipedia, fixed by removing 26 edges).
- Data contamination from Mistral 7B pretraining cannot be controlled.
Experimental setup
Datasets: Wikipedia categories arXiv taxonomy
Benchmarks: Wikipedia categories ontology arXiv taxonomy
Metrics: Literal F1 Fuzzy F1 Continuous F1 Graph F1 Motif distance
Baselines: Memorisation Hearst REBEL Zero-shot prompting One-shot prompting Three-shot prompting Finetune LLMs4OL
Builds on: Hearst patterns REBEL LLMs4OL LoRA spmi
Closest papers by meaning
- Entity Alignment with Noisy Annotations from Large Language Models (NeurIPS 2024 · similarity 0.84)
- HLMEA: Unsupervised Entity Alignment Based on Hybrid Language Models (AAAI 2025 · similarity 0.84)
- Think-on-Graph: Deep and Responsible Reasoning of Large Language Model on Knowledge Graph (ICLR 2024 · similarity 0.84)
- Exploring the Limits of Sub-Billion Language Model Reasoners with Open Training Recipes (ICLR 2026 · similarity 0.83)
- Semantic-KG: Using Knowledge Graphs to Construct Benchmarks for Measuring Semantic Similarity (NeurIPS 2025 · similarity 0.83)
- DiscoveryBench: Towards Data-Driven Discovery with Large Language Models (ICLR 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.