Research map / Vision-language and multimodal / Multimodal LLMs

MME-Unify: A Comprehensive Benchmark for Unified Multimodal Understanding and Generation Models

ICLR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Benchmarks and analysis of MLLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces MME-Unify (MME-U), a benchmark of 4,104 samples evaluating U-MLLMs on understanding, generation, and five mixed-modality unified tasks, reporting scores for 22 models.

Problem

Existing evaluations of unified multimodal LLMs use isolated, inconsistent benchmarks and metrics for understanding and generation, and lack a standardized benchmark for mixed-modality (interleaved text-image) generation tasks.

Approach

Unifies understanding (multiple-choice accuracy), generation (normalized per-task metrics averaged to 0-100), and five unified tasks scored with text accuracy, CLIP-based image-choice accuracy, and a strict all-correct acc+ metric into a single MME-U score.

Key results

• Gemini2.0-flash-exp achieves the highest MME-U score of 45.57, with understanding 65.24, generation 29.79, and unify 40.74.
• Best open-source model MIO-Instruct scores 37.17 overall, while Janus-Pro scores 18.10 and Janus-Flow 16.31.
• No open-source model surpasses 30% accuracy on most unify subtasks, and no model completes any Visual CoT maze successfully (acc+ 0 for most models).

Surprising findings

• Janus-Pro scores zero on unified tasks because it cannot produce interleaved multimodal output, despite strong understanding.
• Gemini2.0-flash-exp outperforms DALL-E-3 by about six points on text-to-image generation under the benchmark's scoring.

Limitations

• Image generation is evaluated via CLIP-based multiple-choice selection, which may allow metric hacking where stylistically poor images score high.
• GPT-4o image generation API was unavailable, so it could not be evaluated for generation.
• Some generation and unify subtasks have small sample sizes (e.g., 52 for auxiliary lines, 50+ per subtask).

Experimental setup

Datasets: MME MMBench MME-RealWorld SEED-Bench-2 Video-MME ImagenHub Emu-Edit TIP-I2V COCO ImageNet MSR-VTT Pexels Videos

Benchmarks: MME-Unify

Metrics: accuracy acc+ FVD FID CLIPSIM LPIPS CLIP-I CLIP-T MME-U score

Baselines: Janus-Pro Janus-Flow Emu3 Show-o VILA-U GILL Anole MIO-Instruct SEED-LLaMA MiniGPT-5 HermesFlow Gemini2.0-flash

Builds on: MME MMBench Emu-Edit MSR-VTT InstructPix2Pix

Closest papers by meaning

• MMIE: Massive Multimodal Interleaved Comprehension Benchmark for Large Vision-Language Models (ICLR 2025 · similarity 0.93)
• WEAVE: Unleashing and Benchmarking the In-context Interleaved Comprehension and Generation (CVPR 2026 · similarity 0.92)
• OpenING: A Comprehensive Benchmark for Judging Open-ended Interleaved Image-Text Generation (CVPR 2025 · similarity 0.91)
• SEED-Bench: Benchmarking Multimodal Large Language Models (CVPR 2024 · similarity 0.91)
• TokenSwap: Benchmarking and Reducing the Modality Gap in Multimodal LLMs (NeurIPS 2026 · similarity 0.90)
• Show-o: One Single Transformer to Unify Multimodal Understanding and Generation (ICLR 2025 · similarity 0.90)