Research map / Image generation / Text-to-image diffusion
MMMG: A Massive, Multidisciplinary, Multi-Tier Generation Benchmark for Text-to-Image Reasoning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces knowledge image generation and the MMMG benchmark (4,456 expert-validated prompt-image pairs with knowledge graphs), a graph-edit-distance plus segmentation-based readability metric (MMMG-Score), and the FLUX-Reason baseline that conditions FLUX.1-dev on reasoning traces.
Problem
Existing text-to-image benchmarks emphasize compositionality, aesthetics, and prompt adherence rather than reasoning-driven generation of explanatory knowledge images (charts, diagrams, mind maps) across disciplines and education levels.
Approach
Represent each knowledge image by a knowledge graph of entities and typed dependencies, score generated images by normalized graph edit distance between grounded and reference KGs, multiplied by a SAM-2.1-based readability penalty on segment count, and train a reasoning-conditioned FLUX variant on CoT traces derived from KGs.
Key results
- GPT-4o achieves an average MMMG-Score of 50.20, the highest among 16 T2I models evaluated.
- FLUX-Reason (R1) reaches 34.45 average MMMG-Score, versus 19.13 for FLUX.1-[dev], an open baseline.
- MMMG-Score reaches Pearson r=0.876 with human ratings, versus 0.701 for WISE and 0.215 for AES-2.5.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add structured reasoning traces to FLUX.1-[dev] generation | FLUX.1-[dev] on MMMG · MMMG | MMMG-Score (×100), Avg: 19.13 -> 34.45 | Helped |
| Use recaptioning instead of structured reasoning traces | FLUX.1-[dev] on MMMG · MMMG | MMMG-Score (×100), Avg: 19.13 -> 18.10 | Hurt |
| Use DeepSeek-R1-Full rather than OpenAI-o3 as the reasoning LLM | FLUX-Reason on MMMG · MMMG | MMMG-Score (×100), Avg: 26.62 -> 34.45 | Helped |
| Use DeepSeek-R1-Full rather than DeepSeek-R1-Distill-Qwen-7B | FLUX-Reason on MMMG · MMMG | MMMG-Score (×100), Avg: 31.26 -> 34.45 | Helped |
| Remove the visual-readability penalty from MMMG-Score (ablation) | MMMG evaluation · MMMG | visual validity of model ranking | Hurt |
| Remove the visual-readability penalty from MMMG-Score (ablation) | Infinity-8B on MMMG · MMMG | 1 - GED (×100), Avg: 38.08 | Mixed |
| Use MMMG-Score instead of WISE for alignment with human ratings | automatic evaluation metrics on MMMG · MMMG human ratings | Pearson correlation: r=0.701 -> r=0.876 | Helped |
Surprising findings
- Omitting the readability penalty lets Infinity-8B score abnormally high (38.08 avg 1-GED), showing knowledge fidelity alone is gameable by cluttered images.
- Weaker models such as LlamaGen and Emu-3 perform best on Geography and Literature, while Economics, History, and Sociology remain hard even for strong models.
Limitations
- OpenAI-o3 struggles to verify dozens of entities and relations in a generated image, and missing items default to false.
- Evaluation depends on proprietary OpenAI-o3 as the grounding judge, so scores are not fully reproducible without API access.
- Collecting more high-quality knowledge images from fragmented sources remains difficult.
Experimental setup
Datasets: MMMG benchmark 16,000 GPT-4o-generated knowledge image training samples
Benchmarks: MMMG
Metrics: MMMG-Score 1-GED readability score FID aesthetic score WISE Pearson correlation with human ratings
Baselines: GPT-4o HiDream-I1-Full FLUX.1-[pro] FLUX.1-[dev] SDXL-1.0 Infinity BAGEL Janus-Pro-7B SimpleAR Emu-3 SEED-X WISE
Builds on: FLUX.1-dev Segment Anything Model 2 WISE chain-of-thought prompting
Closest papers by meaning
- ROVER: Benchmarking Reciprocal Cross-Modal Reasoning for Omnimodal Generation (ICLR 2026 · similarity 0.90)
- GIR-Bench: Versatile Benchmark for Generating Images with Reasoning (ICLR 2026 · similarity 0.90)
- When Visualizing is the First Step to Reasoning: MIRA, a Benchmark for Visual Chain-of-Thought (CVPR 2026 · similarity 0.89)
- Factuality Matters: When Image Generation and Editing Meet Structured Visuals (ICLR 2026 · similarity 0.89)
- MILR: Improving Multimodal Image Generation via Test-Time Latent Reasoning (ICLR 2026 · similarity 0.89)
- VGR: Visual Grounded Reasoning (ICLR 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.