Research map / Vision-language and multimodal / Multimodal LLMs
ScImage: How good are multimodal large language models at scientific text-to-image generation?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces ScImage, a template-generated benchmark of 404 prompts evaluating LLMs and image generators on scientific text-to-image generation via code (Python/TikZ) and direct raster output, scored by 11 scientists on correctness, relevance, and scientific style.
Problem
Multimodal LLMs are strong at general text-to-image generation, but their ability to generate scientific figures (charts, graphs, diagrams) with precise spatial, numeric, and attribute control remains underexplored.
Approach
A structured benchmark built from dictionary and query templates that targets spatial, numeric, and attribute understanding (individually and combined) across scientific object types, compares code-based outputs (Python, TikZ) against direct image generation, and uses expert human evaluation since automatic metrics correlate poorly with human judgments.
Key results
- GPT-4o in TikZ or Python mode scored highest (about 3.5 correctness, 3.5-3.9 scientific style), but still below 4 on all criteria.
- Spatial understanding was the hardest category for code-based models, while numeric understanding was hardest for Stable Diffusion and DALL-E (below 1.8).
- Best automatic metric (PickScore) reached only Kendall 0.26 correlation with human correctness scores, versus 0.75 for human-human agreement.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use GPT-4o rather than other evaluated models for scientific text-to-image generation | ScImage generation · ScImage | correctness: 3.50 (GPT-4o_tikz) and 3.51 (GPT-4o_python); all other models 1.7 to 2.2 | Helped |
| Generate images from code rather than directly from text | ScImage generation · ScImage | scientific style: > 2.5 vs. < 2 | Helped |
| Use Python rather than TikZ as the code output format | GPT-4o and Llama 3.1 8B · ScImage | correctness: 2.64 -> 2.81 | Helped |
| Use Python rather than TikZ as the code output format | GPT-4o and Llama 3.1 8B · ScImage | compile error rate: 0.19 -> 0.17 | Helped |
| Use Python rather than TikZ output | Llama 3.1 8B · ScImage | correctness: 1.78 -> 2.10 | Helped |
| Use Python rather than TikZ output | GPT-4o · ScImage | correctness: 3.50 -> 3.51 | Helped |
| Ignore compile errors when scoring generated code images (ablation) | Llama 3.1 8B_tikz · ScImage | correctness: 1.78 -> 2.49 | Helped |
| Ignore compile errors when scoring generated code images (ablation) | Llama 3.1 8B_python · ScImage | correctness: 2.10 -> 2.92 | Helped |
| Prompt with Chinese rather than English | Evaluated models, multilingual subset · ScImage multilingual | relevance: 3.24 (EN) -> 3.28 (ZH) | Helped |
| Prompt with Chinese rather than English | Evaluated models, multilingual subset · ScImage multilingual | correctness: 3.19 (EN) -> 3.01 (ZH) | Hurt |
| Use English rather than Farsi prompts | Evaluated models, multilingual subset · ScImage multilingual | correctness: 3.19 (EN) -> 2.82 (FA) | Helped |
| Use text-code-image rather than direct text-image generation | Scientific image generation models · ScImage | scientific style: > 2.5 vs. < 2 | Helped |
Surprising findings
- Code-based outputs substantially outperform direct raster generation on scientific style, with DALL-E and Stable Diffusion scoring below 2 on scientificness.
- OpenAI o1-preview beats GPT-4o in English but performs substantially worse in non-English languages such as German and Chinese.
Limitations
- Small sample sizes for some object categories and the multilingual evaluation (20 prompts) limit reliability.
- A single shared prompt was used across all models, so model-specific prompting was not explored, which may disadvantage AutomaTikZ.
- Human evaluation is costly (about 3,000 USD), limiting scalability of the benchmark.
Experimental setup
Datasets: DaTikZ
Benchmarks: ScImage
Metrics: correctness relevance scientific style compile error rate Kendall correlation
Baselines: AutomaTikZ Llama 3.1 8B Stable Diffusion DALL-E OpenAI o1-preview
Builds on: T2I-CompBench DaTikZ AutomaTikZ
Closest papers by meaning
- OpenING: A Comprehensive Benchmark for Judging Open-ended Interleaved Image-Text Generation (CVPR 2025 · similarity 0.90)
- MME-SCI: A Comprehensive and Challenging Science Benchmark for Multimodal Large Language Models (AAAI 2026 · similarity 0.90)
- MME-Unify: A Comprehensive Benchmark for Unified Multimodal Understanding and Generation Models (ICLR 2026 · similarity 0.89)
- LLM Blueprint: Enabling Text-to-Image Generation with Complex and Detailed Prompts (ICLR 2024 · similarity 0.89)
- Exploring the Deep Fusion of Large Language Models and Diffusion Transformers for Text-to-Image Synthesis (CVPR 2025 · similarity 0.88)
- A High Quality Dataset and Reliable Evaluation for Interleaved Image-Text Generation (ICLR 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.