Research map / Image generation / Text-to-image diffusion
Interleaved Scene Graphs for Interleaved Text-and-Image Generation Assessment
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces ISG, a scene-graph-based multi-level evaluation framework for interleaved text-and-image generation, the ISG-Bench benchmark (1,150 samples), and ISG-Agent, a plan-execute-refine compositional baseline.
Problem
Evaluating interleaved text-and-image generation is hard: existing benchmarks are language-dominant, lack golden answers, and rely on coarse MLLM-as-a-judge holistic scoring that misses fine-grained consistency.
Approach
Parse each query into a scene-graph-like structure of text/image blocks and entities, and evaluate responses at four granularities (holistic, structural, block, image) using generated QA pairs with a VQA module, plus golden answers for holistic judging.
Key results
- ISG VQA score reaches Pearson 0.718 with human judgments at block level and 0.907 accuracy at image level.
- Compositional Claude & SD3 averages 6.254 holistic vs 2.961 for best unified model (CoMM-MiniGPT-5); ISG-Agent reaches 6.262.
- ISG-Agent achieves 0.871 structural accuracy vs 0.385 for the best prior model (Gemini).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use compositional generation instead of the strongest unified model | Interleaved generation on ISG-Bench · ISG-Bench holistic | score: 2.961 -> 6.262 | Helped |
| add few-shot examples (ablation) | ISG block-level evaluation · ISG-Bench | accuracy: 0.671 -> 0.942 | Helped |
| add few-shot examples (ablation) | ISG image-level evaluation · ISG-Bench | accuracy: 0.711 -> 0.811 | Helped |
| add vision input (ablation) | ISG block-level evaluation with few-shot examples · ISG-Bench | accuracy: 0.967 -> 0.942 | Hurt |
| add vision input (ablation) | ISG image-level evaluation with few-shot examples · ISG-Bench | accuracy: 0.804 -> 0.811 | Helped |
| remove refinement (ablation) | ISG-Agent · ISG-Bench block | score: 5.366 -> 5.515 | Hurt |
| remove refinement (ablation) | ISG-Agent · ISG-Bench holistic | score: 5.974 -> 6.262 | Hurt |
| remove refinement (ablation) | ISG-Agent · ISG-Bench structural | accuracy: 0.876 -> 0.883 | No clear effect |
| remove refinement (ablation) | ISG-Agent · ISG-Bench image | score: 0.598 -> 0.574 | Mixed |
| use advanced tools instead of non-SOTA tools (ablation) | ISG-Agent · ISG-Bench block | score: 5.155 -> 5.515 | Helped |
| use advanced tools instead of non-SOTA tools (ablation) | ISG-Agent · ISG-Bench image | score: 0.554 -> 0.574 | Helped |
| use advanced tools instead of non-SOTA tools (ablation) | ISG-Agent · ISG-Bench holistic | score: 5.433 -> 6.262 | Helped |
Surprising findings
- MLLM-as-a-Judge shows an image-quality bias, rewarding high-quality images even when they violate instructions.
- CLIPScore failed for block-level evaluation because text blocks exceed the 77-token encoder limit.
Limitations
- Evaluation relies on GPT-4o and MLLMs, which make mistakes and have trustworthiness concerns.
- Vision-dominated tasks like style transfer remain hard to evaluate and generate accurately for all models.
- Unified models often fail to produce the required number of images, preventing block and image level evaluation.
Experimental setup
Datasets: StorySalon UnlearnCanvas TransientAttributes MIT-Adobe FiveK MORPH COCO SA-1B BG-20K DiffMorpher dataset CrossTask ChangeIt Panda-70M
Benchmarks: ISG-Bench
Metrics: Pearson correlation with human judgments structural accuracy holistic score (1-10) block-level VQA score image-level VQA accuracy human agreement
Baselines: Show-o Anole MiniGPT-5 CoMM-MiniGPT-5 Seed-LLaMA-14B Claude & SD3 Gemini & SD3
Builds on: TIFA CLIPScore Chain-of-Thought prompting MLLM-as-a-Judge
Closest papers by meaning
- OpenING: A Comprehensive Benchmark for Judging Open-ended Interleaved Image-Text Generation (CVPR 2025 · similarity 0.90)
- A High Quality Dataset and Reliable Evaluation for Interleaved Image-Text Generation (ICLR 2026 · similarity 0.90)
- Davidsonian Scene Graph: Improving Reliability in Fine-grained Evaluation for Text-to-Image Generation (ICLR 2024 · similarity 0.90)
- Learning Interleaved Image-Text Comprehension in Vision-Language Large Models (ICLR 2025 · similarity 0.88)
- OneIG-Bench: Omni-dimensional Nuanced Evaluation for Image Generation (NeurIPS 2025 · similarity 0.88)
- WEAVE: Unleashing and Benchmarking the In-context Interleaved Comprehension and Generation (CVPR 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.