Research map / Vision-language and multimodal / Multimodal LLMs
GIR-Bench: Versatile Benchmark for Generating Images with Reasoning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces GIR-Bench, a reasoning-centric benchmark with three subsets (understanding-generation consistency, reasoning T2I, reasoning editing) using task-specific deterministic metrics instead of MLLM-as-a-judge, evaluating 21 models.
Problem
Existing benchmarks for unified multimodal models lack reasoning-centric evaluation that measures alignment between understanding and generation, and rely on MLLM-as-a-judge which introduces bias.
Approach
Decomposes reasoning-driven generation and editing into tasks with verifiable ground truth (DINOv3 similarity, object detection counts/boxes, OCR substring score, FID on jigsaw, Sudoku digit accuracy, IoU on segmentation proxies), and pairs understanding and generation on the same entities to measure the gap.
Key results
- GPT-Image-1 achieves the top UGC generation score of 0.689 and Gemini-2.5-Flash-Image the top T2I overall of 0.650, while open-source unified models lag (e.g., Janus-Pro 0.211 UGC).
- BAGEL with chain-of-thought raises numerical reasoning from 0.056 to 0.249 and spatial layout from 0.287 to 0.460 on T2I.
- Understanding accuracy is high (0.874 to 0.974 for unified models) while generation under implicit prompts drops well below category-prompt performance, revealing a persistent understanding-generation gap.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add explicit chain-of-thought reasoning before generation (ablation) | BAGEL-7B · GIR-Bench-UGC Image Generation | Overall score: 0.295 -> 0.341 | Helped |
| add explicit chain-of-thought reasoning before generation (ablation) | BAGEL-7B · GIR-Bench-T2I | Overall score: 0.169 -> 0.276 | Helped |
| add explicit chain-of-thought reasoning before generation (ablation) | BAGEL-7B · GIR-Bench-Edit | Overall score: 0.098 -> 0.140 | Helped |
| add explicit chain-of-thought reasoning before generation (ablation) | BAGEL-7B · GIR-Bench-T2I Text Rendering | Text rendering score: 0.163 -> 0.120 | Hurt |
| use unified model architecture rather than generation-only (overall trend) | unified models vs generation-only models · GIR-Bench-T2I | Overall score | Mixed |
| use explicit task-specific metrics instead of MLLM-as-a-Judge | evaluation protocol · GIR-Bench-T2I and GIR-Bench-Edit | correlation with human judgment: rho ≈ 0.96 for automated metrics | Helped |
Surprising findings
- Open-source unified models show no clear advantage over strong generation-only models on reasoning-centric UGC generation.
- BAGEL with CoT does not improve text rendering over base BAGEL (0.120 vs 0.163).
Limitations
- Evaluation pipelines depend on automated detectors (InternVL3.5-38B grounding, PPOCR), which may have false negatives on stylized or occluded content.
- Edit tasks have low absolute scores and reasoning-perception uses a proxy formulation (green mask) rather than true segmentation.
Experimental setup
Datasets: LISA open-source real-world image datasets
Benchmarks: GIR-Bench-UGC GIR-Bench-T2I GIR-Bench-Edit
Metrics: DINOv3 feature similarity object detection count accuracy bounding box spatial accuracy word-level continuous substring score normalized FID Sudoku digit accuracy IoU
Baselines: GPT-Image-1 Gemini-2.5-Flash-Image BAGEL Qwen-Image FLUX.1-Kontext Step1X-Edit
Builds on: GenEval WISE T2I-ReasonBench RISEBench
Closest papers by meaning
- JourneyBench: A Challenging One-Stop Vision-Language Understanding Benchmark of Generated Images (NeurIPS 2024 · similarity 0.91)
- ROVER: Benchmarking Reciprocal Cross-Modal Reasoning for Omnimodal Generation (ICLR 2026 · similarity 0.91)
- DiffThinker: Towards Generative Multimodal Reasoning with Diffusion Models (ICML 2026 · similarity 0.90)
- GGBench: A Geometric Generative Reasoning Benchmark for Unified Multimodal Models (CVPR 2026 · similarity 0.90)
- IDEA-Bench: How Far are Generative Models from Professional Designing? (CVPR 2025 · similarity 0.90)
- MMMG: A Massive, Multidisciplinary, Multi-Tier Generation Benchmark for Text-to-Image Reasoning (NeurIPS 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.