Research map / Vision-language and multimodal / Multimodal LLMs
SciMKG: A Multimodal Knowledge Graph for Science Education with Text, Image, Video and Audio
Tong Lu, Zhichun Wang, Yaoyu Zhou, Yiming Guan, Zhiyong Bai, Junsheng Du
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SciMKG, an automated framework for building a multimodal science-education knowledge graph covering text, image, video and audio from open educational resources, using LLM reasoning in place of costly manual annotation.
Problem
Constructing multimodal educational knowledge graphs from diverse open educational resources relies on costly manual annotation and lacks multimodal integration.
Approach
An automated pipeline that harnesses LLM reasoning to build a multimodal educational KG integrating text, image, video and audio, avoiding manual annotation. Details beyond the abstract are not available.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use multi-LLM verified stepwise extraction instead of single-LLM or prior NER baselines | SciMKG concept extraction · Concept extraction (100 lessons, 4,479 concepts) | F1: 0.734 (LinkNER) -> 0.803 | Improved |
| add multiple LLMs (voting integration) (ablation) | SciMKG concept extraction · Concept extraction (100 lessons, 4,479 concepts) | recall: 0.342 -> 0.702 | Improved |
| add self-feedback verification (Self-Refine) (ablation) | SciMKG concept extraction · Concept extraction (100 lessons, 4,479 concepts) | precision: 0.374 -> 0.947 | Improved |
| combine multiple LLMs and self-feedback (ablation) | SciMKG concept extraction · Concept extraction (100 lessons, 4,479 concepts) | F1: 0.312 -> 0.803 | Improved |
| use GPT-4o, Gemini-1.5-flash and DeepSeek-V3 instead of small open-source LLM ensemble | SciMKG concept extraction · Concept extraction (100 lessons, 4,479 concepts) | F1: 0.678 -> 0.803 | Improved |
| scale the open-source LLM ensemble from small to large | SciMKG concept extraction · Concept extraction (100 lessons, 4,479 concepts) | F1: 0.678 -> 0.739 | Improved |
| increase verification iterations from 1 to 5 (ablation) | SciMKG concept extraction · Concept extraction (100 lessons, 4,479 concepts) (alpha=0.7) | F1: 0.698 -> 0.803 | Improved |
| raise confidence threshold alpha from 0.7 to 0.9 (ablation) | SciMKG concept extraction · Concept extraction (100 lessons, 4,479 concepts) (iter=5) | F1: 0.803 -> 0.758 | Hurt |
| lower confidence threshold alpha from 0.7 to 0.5 (ablation) | SciMKG concept extraction · Concept extraction (100 lessons, 4,479 concepts) (iter=5) | F1: 0.803 -> 0.792 | Hurt |
Closest papers by meaning
- K12-KGraph: A Curriculum-Aligned Knowledge Graph for Benchmarking and Training Educational LLMs (NeurIPS 2026 · similarity 0.88)
- GraphVis: Boosting LLMs with Visual Knowledge Graph Integration (NeurIPS 2024 · similarity 0.87)
- MME-SCI: A Comprehensive and Challenging Science Benchmark for Multimodal Large Language Models (AAAI 2026 · similarity 0.86)
- APKGC: Noise-enhanced Multi-Modal Knowledge Graph Completion with Attention Penalty (AAAI 2025 · similarity 0.84)
- BMMR: A Large-Scale Bilingual Multimodal Multi-Discipline Reasoning Dataset (NeurIPS 2025 · similarity 0.84)
- Format Matters: The Robustness of Multimodal LLMs in Reviewing Evidence from Tables and Charts (AAAI 2026 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
