AI Paper Atlas

Research map / Vision-language and multimodal / Multimodal LLMs

SciMKG: A Multimodal Knowledge Graph for Science Education with Text, Image, Video and Audio

Tong Lu, Zhichun Wang, Yaoyu Zhou, Yiming Guan, Zhiyong Bai, Junsheng Du

AAAI 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Interleaved image-text generation with MLLMs · 2 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes SciMKG, an automated framework for building a multimodal science-education knowledge graph covering text, image, video and audio from open educational resources, using LLM reasoning in place of costly manual annotation.

Problem

Constructing multimodal educational knowledge graphs from diverse open educational resources relies on costly manual annotation and lacks multimodal integration.

Approach

An automated pipeline that harnesses LLM reasoning to build a multimodal educational KG integrating text, image, video and audio, avoiding manual annotation. Details beyond the abstract are not available.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
use multi-LLM verified stepwise extraction instead of single-LLM or prior NER baselinesSciMKG concept extraction · Concept extraction (100 lessons, 4,479 concepts)F1: 0.734 (LinkNER) -> 0.803Improved
add multiple LLMs (voting integration) (ablation)SciMKG concept extraction · Concept extraction (100 lessons, 4,479 concepts)recall: 0.342 -> 0.702Improved
add self-feedback verification (Self-Refine) (ablation)SciMKG concept extraction · Concept extraction (100 lessons, 4,479 concepts)precision: 0.374 -> 0.947Improved
combine multiple LLMs and self-feedback (ablation)SciMKG concept extraction · Concept extraction (100 lessons, 4,479 concepts)F1: 0.312 -> 0.803Improved
use GPT-4o, Gemini-1.5-flash and DeepSeek-V3 instead of small open-source LLM ensembleSciMKG concept extraction · Concept extraction (100 lessons, 4,479 concepts)F1: 0.678 -> 0.803Improved
scale the open-source LLM ensemble from small to largeSciMKG concept extraction · Concept extraction (100 lessons, 4,479 concepts)F1: 0.678 -> 0.739Improved
increase verification iterations from 1 to 5 (ablation)SciMKG concept extraction · Concept extraction (100 lessons, 4,479 concepts) (alpha=0.7)F1: 0.698 -> 0.803Improved
raise confidence threshold alpha from 0.7 to 0.9 (ablation)SciMKG concept extraction · Concept extraction (100 lessons, 4,479 concepts) (iter=5)F1: 0.803 -> 0.758Hurt
lower confidence threshold alpha from 0.7 to 0.5 (ablation)SciMKG concept extraction · Concept extraction (100 lessons, 4,479 concepts) (iter=5)F1: 0.803 -> 0.792Hurt

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy