Research map / Image generation / Text-to-image diffusion
SCIEval: Evaluating and Benchmarking the Faithfulness of Scientific Image Generation and Interpretation with Large Multimodal Models
Guanghui Ye, Huan Zhao, Zhixue Zhao, Tengfei Ma, Kehan Wang, Steffen Eger, Zhihua Jiang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add cross-modal contrastive loss (ablation) | SCIEval-R/A · Sci-T2I CS relevance | Spearman (%): 53.2 -> 74.1 | Improved |
| add intra-modal contrastive loss (ablation) | SCIEval-R/A · Sci-T2I CS relevance | Spearman (%): 68.4 -> 74.1 | Improved |
| initialize accuracy CLIP from relevance CLIP (ablation) | SCIEval-A · Sci-T2I CS accuracy | Spearman (%): 66.2 -> 69.4 | Improved |
| increase training data from 25% to 100% (ablation) | SCIEval · Sci-IC CS relevance | Spearman (%): 60.5 -> 75.9 | Improved |
| scale model size within InstructBLIP | InstructBLIP · SCIEval-Bench | Spearman (%): Sci-T2I CS rel. 41.1 (7b) vs 41.2 (13b) | No clear effect |
| use SCIEval instead of GPT-4o as evaluator | SCIEval · Sci-T2I CS relevance | Spearman (%): 71.3 -> 74.1 | Improved |
| evaluate CS subset instead of General | SCIEval · Sci-T2I relevance | Spearman (%): 68.5 -> 74.1 | Improved |
| use SCIEval rationale module over baselines | SCIEval-E · rationale completeness (human) | 5-pt score: 3.5 -> 4.2 | Improved |
| use SCIEval rationale module over GPT-4V on correctness (LMM judge) | SCIEval-E · rationale correctness (LMM judge) | 5-pt score: 4.4 -> 4.4 | No clear effect |
Closest papers by meaning
- ScImage: How good are multimodal large language models at scientific text-to-image generation? (ICLR 2025 · similarity 0.90)
- SciFIBench: Benchmarking Large Multimodal Models for Scientific Figure Interpretation (NeurIPS 2024 · similarity 0.87)
- Science-T2I: Addressing Scientific Illusions in Image Synthesis (CVPR 2025 · similarity 0.86)
- PerceptionRubrics: Calibrating Multimodal Evaluation to Human Perception (ICML 2026 · similarity 0.85)
- SciVideoBench: Benchmarking Scientific Video Reasoning in Large Multimodal Models (ICML 2026 · similarity 0.85)
- MMMG: A Massive, Multidisciplinary, Multi-Tier Generation Benchmark for Text-to-Image Reasoning (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
