Research map / Vision-language and multimodal / Multimodal LLMs
ModalSyncSum: Synchronizing Image and Text for Reliable Summary Generation
Xuanqi Chen, Ziying Rong, Xinfeng Liao, Yiqian Wu, Bowei Zhang, Pengfei Fu, Shengyi Jiang
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ModalSyncSum to improve multimodal summarization with multimodal output (MSMO) by synchronizing image and text to reduce hallucination and weak visual-text alignment.
Problem
Multimodal summarization with multimodal output (MSMO) requires coherent text summaries plus semantically relevant image selection; large multimodal models often hallucinate and align visuals and text poorly.
Approach
Synchronizes image and text generation for summarization to improve visual-text alignment and reliability; the abstract names GPT-4o, LLaMA-3, and Grok-3 as the models showing these failures, and the method details are not available in the abstract.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply ModalSyncSum framework instead of direct prompting | GPT-4o · MSMO | BERTScore: 54.37 -> 59.48 | Improved |
| apply ModalSyncSum framework instead of direct prompting | GLM-4v · MSMO | M3AS: 26.18 -> 35.46 | Improved |
| apply ModalSyncSum vs trained multimodal baselines | GLM-4v · MSMO | ROUGE-1: 41.94 (Va-SOGM) -> 39.95 | Hurt |
| apply ModalSyncSum vs trained multimodal baselines | GLM-4v · MSMO | BERTScore: 48.94 (VG-GPLMs) -> 60.71 | Improved |
| add image-description consistency check (CLIP+BLIP QA) (ablation) | ModalSyncSum (GLM-4v) · MSMO | ROUGE-1: 35.18 -> 39.95 | Improved |
| add image-description consistency check (ablation) | ModalSyncSum (GLM-4v) · MSMO | M3AS: 30.07 -> 35.46 | Improved |
| add summary consistency check (NER QA) (ablation) | ModalSyncSum (GLM-4v) · MSMO | M3AS: 33.13 -> 35.46 | Improved |
| add summary consistency check (ablation) | ModalSyncSum (GLM-4v) · MSMO | BERTScore: 55.32 -> 60.71 | Improved |
| pass images directly to GPT-4o instead of text only (ablation) | GPT-4o · MSMO | ROUGE-1: 31.31 -> 27.22 | Hurt |
| raise tau1 from 0.3 to 0.5 (ablation) | ModalSyncSum · MSMO | M3AS: 35.46 -> 25.24 | Hurt |
| lower tau2 filtering threshold from 0.4 to 0.1 (ablation) | ModalSyncSum · MSMO | M3AS: 35.46 -> 31.73 | Hurt |
Experimental setup
Baselines: GPT-4o LLaMA-3 Grok-3
Closest papers by meaning
- DIUSum: Dynamic Image Utilization for Multimodal Summarization (AAAI 2024 · similarity 0.88)
- SILMM: Self-Improving Large Multimodal Models for Compositional Text-to-Image Generation (CVPR 2025 · similarity 0.88)
- CoMM: A Coherent Interleaved Image-Text Dataset for Multimodal Understanding and Generation (CVPR 2025 · similarity 0.87)
- SO-Bench: A Structural Output Evaluation of Multimodal LLM (CVPR 2026 · similarity 0.87)
- MMSum: A Dataset for Multimodal Summarization and Thumbnail Generation of Videos (CVPR 2024 · similarity 0.87)
- OpenING: A Comprehensive Benchmark for Judging Open-ended Interleaved Image-Text Generation (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
