AI Paper Atlas

Research map / Vision-language and multimodal / Multimodal LLMs

ModalSyncSum: Synchronizing Image and Text for Reliable Summary Generation

Xuanqi Chen, Ziying Rong, Xinfeng Liao, Yiqian Wu, Bowei Zhang, Pengfei Fu, Shengyi Jiang

AAAI 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Unified any-modality language models

Read the paperCheck your idea against 137,217 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes ModalSyncSum to improve multimodal summarization with multimodal output (MSMO) by synchronizing image and text to reduce hallucination and weak visual-text alignment.

Problem

Multimodal summarization with multimodal output (MSMO) requires coherent text summaries plus semantically relevant image selection; large multimodal models often hallucinate and align visuals and text poorly.

Approach

Synchronizes image and text generation for summarization to improve visual-text alignment and reliability; the abstract names GPT-4o, LLaMA-3, and Grok-3 as the models showing these failures, and the method details are not available in the abstract.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
apply ModalSyncSum framework instead of direct promptingGPT-4o · MSMOBERTScore: 54.37 -> 59.48Improved
apply ModalSyncSum framework instead of direct promptingGLM-4v · MSMOM3AS: 26.18 -> 35.46Improved
apply ModalSyncSum vs trained multimodal baselinesGLM-4v · MSMOROUGE-1: 41.94 (Va-SOGM) -> 39.95Hurt
apply ModalSyncSum vs trained multimodal baselinesGLM-4v · MSMOBERTScore: 48.94 (VG-GPLMs) -> 60.71Improved
add image-description consistency check (CLIP+BLIP QA) (ablation)ModalSyncSum (GLM-4v) · MSMOROUGE-1: 35.18 -> 39.95Improved
add image-description consistency check (ablation)ModalSyncSum (GLM-4v) · MSMOM3AS: 30.07 -> 35.46Improved
add summary consistency check (NER QA) (ablation)ModalSyncSum (GLM-4v) · MSMOM3AS: 33.13 -> 35.46Improved
add summary consistency check (ablation)ModalSyncSum (GLM-4v) · MSMOBERTScore: 55.32 -> 60.71Improved
pass images directly to GPT-4o instead of text only (ablation)GPT-4o · MSMOROUGE-1: 31.31 -> 27.22Hurt
raise tau1 from 0.3 to 0.5 (ablation)ModalSyncSum · MSMOM3AS: 35.46 -> 25.24Hurt
lower tau2 filtering threshold from 0.4 to 0.1 (ablation)ModalSyncSum · MSMOM3AS: 35.46 -> 31.73Hurt

Experimental setup

Baselines: GPT-4o LLaMA-3 Grok-3

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy