Research map / Vision-language and multimodal / Multimodal LLMs

A High Quality Dataset and Reliable Evaluation for Interleaved Image-Text Generation

ICLR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Interleaved image-text generation with MLLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces InterSyn, a 1.8M-sample interleaved image-text instruction dataset built with an automated Generate-Evaluate-Refine pipeline (SEIR), and SynJudge, a fine-tuned MLLM judge scoring TCC, ICC, IQ, and ITS.

Problem

Interleaved image-text generation by LMMs is limited by small, low-quality, and instructionally poor training data, and existing evaluators correlate poorly with human judgment.

Approach

A Generate-Evaluate-Refine loop (SEIR) with a judge model iteratively refines questions, answers/captions, and images to build a large dataset, and a fine-tuned QwenVL judge (SynJudge) provides four interpretable scores including an image-text synergy metric.

Key results

• SynJudge (QwenVL-trained) reaches the highest average A@1 of 95.4%, versus about 86.5% for zero-shot GPT-4o.
• SEIR-generated InterSyn samples outperform the strongest baseline GPT-4o+DALL-E3 by 0.34-0.66 on mean scores across dimensions in human evaluation.
• Fine-tuning Anole on 25k-50k InterSyn samples raises ITS from 2.26 to 2.40 (25k) and 2.74 (50k) under SynJudge scoring; 200k reaches 3.11.

Surprising findings

• Zero-shot SynJudge-style judges such as GPT-4o show markedly lower human agreement than fine-tuned judges, and the paper reports fine-tuned judge gains as decisive.
• Fine-tuning generators on InterSyn causes small drops on some standard understanding benchmarks (e.g., BAGEL MMBench -1.9, MMMU -2.5).

Limitations

• Judge training and evaluation rely on a single human-annotated set of 48,000 pairs with human scoring by ten experts.
• Reliance on proprietary or synthetic generators (GPT-4o, DALL-E3) for some baselines and data generation.

Experimental setup

Datasets: InterSyn human-annotated SynJudge data (48,000 pairs)

Benchmarks: InterSyn evaluation benchmark (4,000 questions) MME-P MMBench MMMU SEEDBench

Metrics: mean score variance RMSE A@1 (agreement within 1 point)

Baselines: Emu3 Janus-Pro VILA-U Show-o Show-o-Turbo Liquid D-DiT Anole VARGPT VARGPT-v1.1 BAGEL GPT-4o+DALL-E3

Builds on: SEIR-style refinement LeafInstruct Janus-Pro

Closest papers by meaning

• CoMM: A Coherent Interleaved Image-Text Dataset for Multimodal Understanding and Generation (CVPR 2025 · similarity 0.93)
• OpenING: A Comprehensive Benchmark for Judging Open-ended Interleaved Image-Text Generation (CVPR 2025 · similarity 0.92)
• MMIE: Massive Multimodal Interleaved Comprehension Benchmark for Large Vision-Language Models (ICLR 2025 · similarity 0.91)
• OmniCorpus: A Unified Multimodal Corpus of 10 Billion-Level Images Interleaved with Text (ICLR 2025 · similarity 0.90)
• Interleaved Scene Graphs for Interleaved Text-and-Image Generation Assessment (ICLR 2025 · similarity 0.90)
• WEAVE: Unleashing and Benchmarking the In-context Interleaved Comprehension and Generation (CVPR 2026 · similarity 0.89)