Research map / Vision-language and multimodal / Multimodal LLMs
CoMM: A Coherent Interleaved Image-Text Dataset for Multimodal Understanding and Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces CoMM, a 227K-document coherent interleaved image-text dataset built from instructional and visual-story sites with multi-perspective LLM/CLIP/GPT-4o filtering, plus four interleaved generation tasks and GPT-4o-based evaluation.
Problem
Interleaved image-text generation by MLLMs lacks narrative coherence and entity/style consistency, which the paper attributes to low-quality interleaved training data.
Approach
Curate interleaved documents from instructional (WikiHow, eHow, Instructables) and visual-story sites, filter them with a multi-perspective pipeline (LLM text-development filter, CLIP image-sequence coherence metric, CLIP plus MLLM image-text alignment filter), and derive a preference dataset by shuffling steps to create negatives.
Key results
- Open-Flamingo 9B few-shot with CoMM reaches COCO CIDEr 109.3 at 16 shots vs 98.8 baseline and 102.9 with MMC4.
- On dataset quality by GPT-4o, CoMM scores 7.64 development, 7.07 completeness, 8.91 image-text alignment vs MMC4 4.75/5.12/4.66.
- SEED-Llama-8B DPO on the preference data raises Task 1 METEOR (step) from 2.29 to 4.09.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train Open-Flamingo 9B with CoMM rather than the strongest listed dataset baseline | Open-Flamingo 9B · COCO test, 16-shot | CIDEr: 100.8 -> 109.3 | Helped |
| Train Open-Flamingo 9B with CoMM rather than the strongest listed dataset baseline | Open-Flamingo 9B · Flickr30k test (Karpathy), 16-shot | CIDEr: 65.5 -> 71.5 | Helped |
| Train Open-Flamingo 9B with CoMM rather than no additional dataset training | Open-Flamingo 9B · VQAv2 testdev, 16-shot | VQA accuracy: 54.3 -> 54.2 | Hurt |
| Train Open-Flamingo 9B with CoMM rather than the strongest listed dataset baseline | Open-Flamingo 9B · OKVQA val, 16-shot | VQA accuracy: 45.3 -> 46.9 | Helped |
| Train Open-Flamingo 9B with CoMM rather than the strongest listed dataset baseline | Open-Flamingo 9B · TextVQA val, 16-shot | VQA accuracy: 30.2 -> 35.8 | Helped |
| Train Open-Flamingo 9B with CoMM rather than no additional dataset training | Open-Flamingo 9B · VizWiz testdev, 16-shot | VQA accuracy: 42.5 -> 43.5 | Helped |
| Train Open-Flamingo 9B with CoMM rather than the strongest listed dataset baseline | Open-Flamingo 9B · HatefulMemes test seen, 16-shot | ROC-AUC: 57.8 -> 59.9 | Helped |
| Use CoMM instead of MMC4 or OBELICS | Interleaved image-text dataset quality · Dataset quality, GPT-4o evaluation | Image-text alignment (ITA): 4.66 / 3.81 -> 8.91 | Helped |
| Add DPO training on CoMM preference data | SEED-Llama 8B · Image-to-text sequence generation (Task 1), step-level | METEOR: 2.29 -> 4.09 | Helped |
| Add DPO training on CoMM preference data | SEED-Llama 8B · Text-to-image sequence generation (Task 2) | FID: 57.96 -> 63.05 | Hurt |
| Set the ITA filtering threshold to 4 instead of 2 (ablation) | CoMM training data · COCO test, 16-shot | CIDEr: 103.1 -> 109.3 | Helped |
| Set the ITA filtering threshold to 4 instead of 2 (ablation) | CoMM training data · Flickr30K, 16-shot | CIDEr: 69.3 -> 71.5 | Helped |
Surprising findings
- Image-sequence coherence metric on MMC4 and OBELICS is far below CoMM (0.21 and 1.00 vs 4.27).
- DPO improves text generation (Task 1) but degrades image generation metrics such as FID and IRS in Task 2 and Task 3.
Limitations
- Dataset scale (2.28M images) is limited relative to web-scale corpora; authors cannot expand to billions due to cost.
- GPT-4o-based evaluation is stricter with humans, and evaluation relies on MLLM judges that may be biased.
- DPO gains on image generation quality are limited.
Experimental setup
Datasets: WikiHow eHow Instructables StoryBird StoryGen MMC4 OBELICS COCO Flickr30k VQAv2 OKVQA TextVQA
Benchmarks: COCO Flickr30k VQAv2 OKVQA TextVQA VizWiz HatefulMemes
Metrics: CIDEr VQA accuracy ROC-AUC METEOR ROUGE_L FID IS SSIM PSNR IRS
Baselines: MMC4 OBELICS MiniGPT-5 SEED-Llama Emu2
Builds on: Open-Flamingo SEED-Llama DPO MMC4
Closest papers by meaning
- A High Quality Dataset and Reliable Evaluation for Interleaved Image-Text Generation (ICLR 2026 · similarity 0.93)
- OpenING: A Comprehensive Benchmark for Judging Open-ended Interleaved Image-Text Generation (CVPR 2025 · similarity 0.92)
- OmniCorpus: A Unified Multimodal Corpus of 10 Billion-Level Images Interleaved with Text (ICLR 2025 · similarity 0.91)
- DreamLLM: Synergistic Multimodal Comprehension and Creation (ICLR 2024 · similarity 0.90)
- Towards Unified Multimodal Interleaved Generation via Group Relative Policy Optimization (NeurIPS 2025 · similarity 0.90)
- MMIE: Massive Multimodal Interleaved Comprehension Benchmark for Large Vision-Language Models (ICLR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.