Research map / Vision-language and multimodal / Multimodal LLMs
SciFIBench: Benchmarking Large Multimodal Models for Scientific Figure Interpretation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces SciFIBench, a 2000-question multiple-choice benchmark (Figure→Caption and Caption→Figure) built from arXiv figure-caption pairs with adversarial negatives and human verification, and evaluates 28 LMMs plus VLM and human baselines.
Problem
Scientific figure interpretation by large multimodal models is poorly characterised, and generating objective ground truth for figure descriptions at scale is difficult.
Approach
Reframes figure understanding as multiple-choice matching between figures and captions, using CLIP-based nearest-neighbour adversarial negatives plus human verification of every question, and uses Gemini-Pro to parse free-form LMM outputs into answer letters.
Key results
- GPT-4o reaches 73.8% overall on Figure→Caption (adversarial negatives, CS and General combined) and Gemini 1.5 Pro reaches 66.1% on Caption→Figure, while the mean human baseline scores 86.4% and 78.4% on a 25-question-per-task subset.
- Closed-source models outperform open-source ones by 34.6% on Figure→Caption at the best-vs-best comparison, and the best open-source model (TransCore-M) underperforms the weakest closed model.
- Performance drops by large margins when adversarial negatives are replaced with random ones, e.g. open-source models more than double accuracy, while open-source models on Caption→Figure show near-chance performance in both settings.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use randomly selected rather than adversarial negatives (ablation) | GPT-4o · SciFIBench CS Figure→Caption | accuracy: 75.4 -> 99.6 | Helped |
| use randomly selected rather than adversarial negatives (ablation) | Gemini 1.5 Flash · SciFIBench CS Figure→Caption | accuracy: 74.4 -> 99.0 | Helped |
| use randomly selected rather than adversarial negatives (ablation) | GPT-4o · SciFIBench CS Caption→Figure | accuracy: 72.2 -> 99.0 | Helped |
| use randomly selected rather than adversarial negatives (ablation) | Gemini 1.5 Pro · SciFIBench CS Caption→Figure | accuracy: 76.0 -> 98.4 | Helped |
| select hard negatives using joint-embedding neighbours rather than single-modality neighbours (ablation) | LMMs on SciFIBench · SciFIBench question difficulty | accuracy | Hurt |
| add few-shot examples from noisy data (ablation) | Gemini 1.5 Flash · SciFIBench | accuracy | Mixed |
| add few-shot examples from noisy data (ablation) | Gemini 1.5 Flash · SciFIBench | accuracy | Hurt |
| mark the correct caption as <Correct> (ablation) | evaluated LMMs · SciFIBench CS Figure→Caption augmented questions | accuracy | Helped |
| mark an incorrect caption as <Correct> (ablation) | evaluated LMMs · SciFIBench CS Figure→Caption augmented questions | accuracy | Hurt |
| instruct the model to ignore the <Correct> annotation (ablation) | evaluated LMMs · SciFIBench CS Figure→Caption augmented questions | accuracy | Mixed |
Surprising findings
- Performance does not necessarily scale with model size; smaller checkpoints often beat larger ones on adversarial negatives.
- A CLIP ViT-H-14 embedding baseline performs comparably to or better than leading open-source LMMs on adversarial negatives.
Limitations
- Benchmark is small (2000 questions) and limited to multiple-choice format, which may not reflect open-ended figure interpretation.
- Human baseline is small (5 participants, 25 questions per task) and not domain experts; human verification may bias toward easier questions.
- Possible leakage of arXiv data into closed-source training sets cannot be ruled out.
Experimental setup
Datasets: SciCap ArXivCap
Benchmarks: SciFIBench
Metrics: accuracy refusal rate caption ranking
Baselines: GPT-4V GPT-4 Turbo GPT-4o Gemini Pro Vision Gemini 1.5 Pro Gemini 1.5 Flash Claude 3 Opus Claude 3 Sonnet Claude 3 Haiku CLIP ViT-H-14-378-quickgelu
Builds on: SciCap ArXivCap CLIP MMMU
Closest papers by meaning
- MRAG-Bench: Vision-Centric Evaluation for Retrieval-Augmented Multimodal Models (ICLR 2025 · similarity 0.88)
- ZeroBench: An Impossible Visual Benchmark for Contemporary Large Multimodal Models (ICML 2026 · similarity 0.88)
- MMIE: Massive Multimodal Interleaved Comprehension Benchmark for Large Vision-Language Models (ICLR 2025 · similarity 0.87)
- SciVideoBench: Benchmarking Scientific Video Reasoning in Large Multimodal Models (ICML 2026 · similarity 0.87)
- ARK: A Dual-Axis Multimodal Retrieval Benchmark along Reasoning and Knowledge (NeurIPS 2026 · similarity 0.87)
- ScImage: How good are multimodal large language models at scientific text-to-image generation? (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.