Research map / Vision-language and multimodal / Human and embodied modeling
UKnow: A Unified Knowledge Protocol with Multimodal Knowledge Graph Datasets for Reasoning and Vision-Language Pre-Training
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes UKnow, a unified knowledge protocol that organizes multimodal data as a knowledge graph with five knowledge-view unit types, and releases a 1.39M-node international-news multimodal KG dataset evaluated on link prediction, event classification, retrieval, and VTAB adaptation.
Problem
Existing multimodal knowledge graph datasets are small, focus on a single task, and lack logical event connections, limiting knowledge-based vision-language pre-training and reasoning.
Approach
A data-centric protocol that categorizes knowledge into five unit types (in-image, in-text, cross-image, cross-text, image-text), a three-phase pipeline (content extraction, information symbolization, knowledge construction) to convert any image-text collection into a multimodal KG, and a news-event KG dataset built with it.
Key results
- BETAE* reaches 21.64% validation MRR and 21.23% test MRR on the UKnow link prediction benchmark, versus 20.75% for BETAE.
- TCL with image input reaches 66.80% ACC@1 on Event-11 and 55.87% on Event-9185 in multimodal event classification.
- CLIP-UKnow pre-training improves VTAB average accuracy to 75.23% versus 74.09% for CLIP* pre-trained on the same data.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add the UKnow plug-in module to Q2B (ablation) | Q2B · UKnow validation and test | MRR: 18.80 -> 19.66 (validation); 18.46 -> 19.27 (test) | Helped |
| Add the UKnow plug-in module to BETAE (ablation) | BETAE · UKnow validation and test | MRR: 21.16 -> 21.64 (validation); 20.75 -> 21.23 (test) | Helped |
| Use image and text inputs together instead of image-only input | TCL · Event-11 and Event-9185 | ACC@1: 66.80 -> 68.69 (Event-11); 55.87 -> 60.89 (Event-9185) | Helped |
| Fine-tune pre-trained parameters instead of using zero-shot inference | TCL · Image retrieval | R@1: 33.24 -> 58.89 | Helped |
| Use UKnow pre-training instead of original CLIP contrastive pre-training on the same dataset | CLIP · VTAB (avg.) | ACC@1: 74.09 -> 75.23 | Helped |
Surprising findings
- Text-only models outperform image-only models on Event-9185 classification (e.g., CLIP 57.48% text vs 54.62% image ACC@1).
Limitations
- Dataset is restricted to international news from Wikipedia Current Events, limiting domain coverage.
- Dataset license is non-commercial (CC BY-NC-ND 4.0), restricting use.
- Event-level logic is absent for image-text pair inputs such as LAION-5B, so only partial knowledge types can be built.
Experimental setup
Datasets: UKnow (international news multimodal KG, from Wikipedia Current Events) LAION-5B VTAB
Benchmarks: UKnow link prediction UKnow event classification (Event-11, Event-9185) UKnow retrieval VTAB
Metrics: MRR Hits@1 Hits@3 Hits@10 ACC@1 ACC@5 R@1 R@5 R@10
Baselines: TransE Q2B BETAE QA-GNN CLIP DeCLIP ALBEF TCL
Builds on: CLIP Q2B BETAE TCL TransE
Closest papers by meaning
- GraphVis: Boosting LLMs with Visual Knowledge Graph Integration (NeurIPS 2024 · similarity 0.89)
- MMFineReason: Closing the Multimodal Reasoning Gap via Open Data-Centric Methods (NeurIPS 2026 · similarity 0.89)
- JourneyBench: A Challenging One-Stop Vision-Language Understanding Benchmark of Generated Images (NeurIPS 2024 · similarity 0.88)
- LENS: Multi-level Evaluation of Multimodal Reasoning with Large Language Models (ICLR 2026 · similarity 0.88)
- Visual Graph Arena: Evaluating Visual Conceptualization of Vision and Multimodal Large Language Models (ICML 2025 · similarity 0.88)
- LION: Empowering Multimodal Large Language Model with Dual-Level Visual Knowledge (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.