Research map / Image generation / Text-to-image diffusion
MDMR-Bench: A Multi-Dimensional Benchmark for Multi-Reference Image Generation
Bowen Zheng, Jintao Lin, Chang-Heng Yi, HAORAN YANG, Binxiao Huang, Chenyang Lei, Rui Liu, Xihui Liu
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces MDMR-Bench, a multi-dimensional benchmark for evaluating multi-reference image generation models conditioned on multiple reference images and text instructions.
Problem
Existing benchmarks for multi-reference image generation rely on limited evaluation dimensions and fail to capture the diverse challenges of conditioning on multiple reference images with textual instructions.
Approach
A benchmark that evaluates multi-reference image generation along multiple dimensions (e.g., entity fusion and attribute transfer across images), in place of the narrower evaluation axes of prior benchmarks.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use checklist-based instead of score-based evaluation (ablation) | MDMR-Bench evaluation protocol · MDMR-Bench judges | inter-judge agreement (Krippendorff's alpha) | Improved |
| increase number of reference images from two to four (ablation) | multi-reference generation models · Attribute Transfer subset | Overall score (0-1 scale): lower by roughly 0.106 (Composition roughly 0.122) | Hurt |
| swap real reference images for synthetic ones (ablation) | Nano Banana Pro · 100 paired cases | Overall (%): 87.01 -> 87.44 | No clear effect |
| swap real reference images for synthetic ones (ablation) | Seedream 5.0 · 100 paired cases | Overall (%): 82.93 -> 83.16 | No clear effect |
| switch judge from Gemini-3-Pro to GPT-5.4 (ablation) | MDMR-Bench leaderboard · MDMR-Bench | model ranking | No clear effect |
| pack multiple elements in a single reference image (multi-element extraction) versus several separate references | multi-reference generation models · MDMR-Bench task families | Overall checklist score | Improved |
| infographic manipulation instead of entity fusion tasks | all models except Nano Banana Pro · MDMR-Bench task families | Overall checklist score | Hurt |
| human-related attribute transfer (expression, makeup, outfit, pose, hair) versus other content types | multi-reference generation models · Attribute Transfer subset | Overall checklist score | Hurt |
Experimental setup
Benchmarks: MDMR-Bench
Closest papers by meaning
- ImagenHub: Standardizing the evaluation of conditional image generation models (ICLR 2024 · similarity 0.88)
- OneIG-Bench: Omni-dimensional Nuanced Evaluation for Image Generation (NeurIPS 2025 · similarity 0.88)
- MMIG-Bench: Towards Comprehensive and Explainable Evaluation of Multi-Modal Image Generation Models (NeurIPS 2025 · similarity 0.88)
- IDEA-Bench: How Far are Generative Models from Professional Designing? (CVPR 2025 · similarity 0.87)
- GIR-Bench: Versatile Benchmark for Generating Images with Reasoning (ICLR 2026 · similarity 0.87)
- MJ-Bench: Is Your Multimodal Reward Model Really a Good Judge for Text-to-Image Generation? (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
