Research map / Graph learning / Knowledge and temporal graphs
Beyond Entities: A Large-Scale Multi-Modal Knowledge Graph with Triplet Fact Grounding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds ImgFact, a large-scale multi-modal KG of 247,732 DBpedia triplet facts grounded to 3.7M Google images via a five-step pipeline (filtering, retrieval, CLIP-prompt entity matching, contrastive relation matching, clustering), and shows gains on link prediction and relation classification.
Problem
Existing multi-modal knowledge graphs ground only entities to images, ignoring the visual semantics of relations; grounding triplet facts (head, relation, tail) to images is unaddressed, and naive entity-grounding pipelines cannot handle non-visualizable triplets or relation matching.
Approach
Grounds whole triplet facts rather than entities: filters non-visualizable entities/relations with a trained classifier and confidence/support metrics, matches entity pairs to images using CLIP with learnable prompts, and matches relations to images via contrastive learning over image titles against triplet sentences.
Key results
- Human evaluation of 500 triplets: 80.6% accuracy, 83.2% H@1, 92.4% H@3 for ImgFact images.
- Relation classification with BERT+ResNet50 using the ImgFact tail image improves weighted F1 by 8.38% over MMKG images and 9.87% over VisualChatGPT images.
- ImgFact contains 247,732 triplets, 182,589 entities, 64 relations and 3,730,805 images.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add ImgFact entity-pair/relation images to BERT+ResNet50 classifier | link prediction (tail) · DL link prediction | MRR: 22.64 to 26.65 (p) | Helped |
| add random noise image instead of matched image (ablation) | link prediction and relation classification · DL | MRR / w-F1: MRR 22.64 to 18.62; w-F1 44.00 to 27.23 | Hurt |
| add ImgFact tail image pt to BERT+ResNet50 for relation classification | relation classification · DL | w-F1: 73.16 vs 64.78 (MMKG) and 63.29 (VisualChatGPT), i.e. 8.38 and 9.87 points relative per paper | Helped |
| add ImgFact relation image pr vs. VisualChatGPT relation images (ablation) | link prediction · DL | MRR: 25.34 (ImgFact pr) vs 24.33 (VCG p*r) | Helped |
| add MMKG or VisualChatGPT entity images in addition to ImgFact images (ablation) | link prediction · DL | MRR: adding p'h to BERT+ResNet50 improves MRR 22.64 to 23.16; adding pr alone gives 25.34, p'h & pr gives 26.03 | Mixed |
| ImgFact relation images added to ViLT classifier for link prediction (ablation) | ViLT link prediction · DL | MRR: ViLT + p'h 22.87, ViLT + pr 23.04, ViLT + p'h & pr 24.56 vs BERT 22.64 | Mixed |
Surprising findings
- Images generated by VisualChatGPT carry relation information but suffer from distortion errors, yet still help relation classification only modestly.
Limitations
- Image noise from outdated DBpedia triplets
- Entity missing in retrieved images
- Entity-matching classifier accuracy only 82%; non-commercial CC-NC license restricts use
Experimental setup
Datasets: DBpedia Google Image Search results MMKG (Liu et al. 2019) DB15K
Benchmarks: ImgFact link prediction split DL ImgFact relation classification split DL
Metrics: H@1 H@10 MRR Accuracy weighted F1 human accuracy pixel-level diversity similarity
Baselines: BERT ViLT MMKG (Liu et al. 2019) VisualChatGPT IMGpedia Richpedia VisualSem
Builds on: CLIP DBpedia prefix tuning DBSCAN BERT
Closest papers by meaning
- Tokenization, Fusion, and Augmentation: Towards Fine-grained Multi-modal Entity Representation (AAAI 2025 · similarity 0.87)
- Multiple Heads are Better than One: Mixture of Modality Knowledge Experts for Entity Representation Learning (ICLR 2025 · similarity 0.87)
- LAFA: Multimodal Knowledge Graph Completion with Link Aware Fusion and Aggregation (AAAI 2024 · similarity 0.87)
- A Dual-Way Enhanced Framework from Text Matching Point of View for Multimodal Entity Linking (AAAI 2024 · similarity 0.86)
- APKGC: Noise-enhanced Multi-Modal Knowledge Graph Completion with Attention Penalty (AAAI 2025 · similarity 0.85)
- Tackling Uncertain Correspondences for Multi-Modal Entity Alignment (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.