Research map / Vision-language and multimodal / Human and embodied modeling
DIUSum: Dynamic Image Utilization for Multimodal Summarization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DIUSum, a multimodal summarization framework with an image selector that scores whether each image is summary-effective and uses hard and soft guidance to plug images into the decoder.
Problem
Existing multimodal summarization fuses all images into encoding regardless of whether each image improves the summary, so redundant or harmful images can degrade quality.
Approach
An image selector trained with self-labels (comparing ROUGE-like rewards of text-only vs multimodal generation) predicts whether an image is summary-effective; its hard decision and soft score gate the image representation fed to the decoder in a multi-stage training scheme.
Key results
- On MMS, DIUSum reaches 50.39 ROUGE-1 versus 49.20 for BertAbs (+1.19).
- On MSMO (Single Img), DIUSum reaches 42.23 ROUGE-1 versus 41.85 for BertAbs (+0.38).
- The image selector achieves F1 of 0.97 on MMS and 0.75 on MSMO in predicting summary-effective images.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use DIUSum dynamic image utilization instead of BertAbs | BertAbs · MMS | ROUGE-1: 49.20 -> 50.39* | Helped |
| use DIUSum dynamic image utilization instead of BertAbs | BertAbs · MSMO (Single Img) | ROUGE-1: 41.85 -> 42.23 | Helped |
| feed images uniformly rather than using text-only input | BertAbs · MMS | ROUGE-1: 49.14 -> 49.20 | No clear effect |
| feed images uniformly rather than using text-only input | BertAbs · MSMO (Single Img) | ROUGE-1: 41.92 -> 41.85 | Hurt |
| add soft guidance to image plugging (ablation) | DIUSum · MMS | ROUGE-1: 49.88 -> 50.39 | Helped |
| add hard guidance to image plugging (ablation) | DIUSum · MMS | ROUGE-1: 50.09 -> 50.39 | Helped |
| add Stage 1 model initialization (ablation) | DIUSum · MMS | ROUGE-1: 49.44 -> 50.39 | Helped |
| keep Stage 2 and Stage 3 separate (ablation) | DIUSum · MMS | ROUGE-1: 48.88 -> 50.39 | Helped |
Surprising findings
- On MSMO (Single Img), text-only BertAbs-txt outperforms the multimodal BertAbs, suggesting most images harm performance.
- Dynamic oracle selection between text-only and multimodal inputs gives large upper-bound gains (49.16 to 54.64 ROUGE-1 on MMS).
Limitations
- Gains on MSMO are small (+0.38 ROUGE-1) because many image-summary pairs lack strong correlation.
- Ground-truth summary-effectiveness labels rely on a self-labeling heuristic using edit distance of first k tokens rather than a true quality measure.
- Evaluated only on two datasets with a single image per sample.
Experimental setup
Datasets: MMS MSMO
Benchmarks: MMS MSMO (Single Img)
Metrics: ROUGE-1 ROUGE-2 ROUGE-L BLEU BERTScore MoverScore ED-top5
Baselines: Lead SEASS Doubly-Attn MAtt CFSum BertAbs-txt BertAbs PGN MSMG MMRank
Builds on: BERT VGG-19 BertAbs CFSum
Closest papers by meaning
- MMSum: A Dataset for Multimodal Summarization and Thumbnail Generation of Videos (CVPR 2024 · similarity 0.85)
- V2Xum-LLM: Cross-Modal Video Summarization with Temporal Prompt Instruction Tuning (AAAI 2025 · similarity 0.83)
- Tell Me What Is Good about This Property: Leveraging Reviews for Segment-Personalized Image Collection Summarization (AAAI 2024 · similarity 0.83)
- OmniCorpus: A Unified Multimodal Corpus of 10 Billion-Level Images Interleaved with Text (ICLR 2025 · similarity 0.83)
- FG-CLIP: Fine-Grained Visual and Textual Alignment (ICML 2025 · similarity 0.82)
- A High Quality Dataset and Reliable Evaluation for Interleaved Image-Text Generation (ICLR 2026 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.