Research map / Vision-language and multimodal / Multimodal LLMs

OpenING: A Comprehensive Benchmark for Judging Open-ended Interleaved Image-Text Generation

CVPR 2025 · Vision-language and multimodal › Multimodal LLMs · approach: Benchmarks and analysis of MLLMs · 6 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces OpenING, a 5,400-instance human-annotated benchmark across 56 tasks for interleaved image-text generation, plus IntJudge, a Qwen2-VL-7B-based pairwise judge trained with arena and RAG-generated data that agrees with humans more than GPT-4o.

Problem

Existing benchmarks for interleaved image-text generation are small and narrow, and GPT-based judges are biased and costly, so there is no reliable, scalable way to evaluate open-ended interleaved generation.

Approach

Combine a broad, human-annotated multi-domain benchmark with a fine-tuned open judge model (IntJudge) trained on pairwise arena labels plus reference-augmented generation (RAG) pairs, evaluated via a roulette-style pairwise arena with tie-handling metrics.

Key results

• IntJudge-7B reaches 82.42% average agreement with human judgments under FDT, versus 71.08% for GPT-4o.
• With RAG data, IntJudge's FDT agreement on unseen models rises by 7.8% over arena-only training (6,014 samples).
• GPT-4o+DALL-E3 and Gemini1.5+Flux are the strongest integrated pipelines, while end-to-end models like MiniGPT-5, GILL and NExT-GPT rank lowest in human win rates.

Measured effects reported in the paper

Surprising findings

• GPT-generated text can be more informative than human-annotated text, while human-annotated natural images remain preferred over generated images.
• GPT-4o judge shows self-preference bias, scoring its own outputs 10 on human preference alignment versus about 9 for human-annotated responses.

Limitations

• GPT-based judges have high error rates and bias; IntJudge is evaluated only against human judgments on a limited set
• Some models (e.g., GPT-4o+DALL-E3) show policy-restricted refusals affecting no-image ratios

Experimental setup

Datasets: OpenING Dev Set

Benchmarks: OpenING Test Set

Metrics: win rate agreement rate with human judgments FDT w/o Tie w/ Tie (0) w/ Tie (0.5)

Baselines: GPT-4o judge InternLM-XComposer2.5-7B judge Qwen2-VL-7B judge Random Anole SEED-X Emu2 Show-o NExT-GPT MiniGPT-5 GILL SEED-LLaMA

Builds on: Qwen2-VL InternLM-XComposer2.5

Closest papers by meaning

• A High Quality Dataset and Reliable Evaluation for Interleaved Image-Text Generation (ICLR 2026 · similarity 0.92)
• CoMM: A Coherent Interleaved Image-Text Dataset for Multimodal Understanding and Generation (CVPR 2025 · similarity 0.92)
• WEAVE: Unleashing and Benchmarking the In-context Interleaved Comprehension and Generation (CVPR 2026 · similarity 0.91)
• MMIE: Massive Multimodal Interleaved Comprehension Benchmark for Large Vision-Language Models (ICLR 2025 · similarity 0.91)
• OmniCorpus: A Unified Multimodal Corpus of 10 Billion-Level Images Interleaved with Text (ICLR 2025 · similarity 0.91)
• MME-Unify: A Comprehensive Benchmark for Unified Multimodal Understanding and Generation Models (ICLR 2026 · similarity 0.91)