Research map / Vision-language and multimodal / Visual question answering
SPIQA: A Dataset for Multimodal Question Answering on Scientific Papers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces SPIQA, a 270K-question multimodal QA dataset over figures and tables from 25.9K computer science arXiv papers, with three evaluation tasks and an LLM log-probability metric (L3Score), benchmarking 12 MLLMs.
Problem
Existing scientific paper QA datasets are small and text-only, ignoring figures and tables that carry key results; models lack evaluation for reasoning over multiple figures, tables, and full paper text.
Approach
Automatically generate large-scale figure/table QA pairs with Gemini 1.5 Pro from paper TeX sources and captions, plus filtered human-written questions from QASA and QASPER, and score free-form answers with L3Score, a yes/no log-probability-based LLM judge metric.
Key results
- GPT-4o achieves the best L3Score on test-A (64.00) and test-C (46.68) for direct QA with figures and tables, while Claude 3 Opus reaches 49.54 on test-B.
- Fine-tuning InstructBLIP-7B and LLaVA-1.5-7B with LoRA on SPIQA training raises average L3Score by about 28 and 26 points over zero-shot, respectively.
- CoT QA with GPT-4o retrieves the correct figure with 86.58% top-1 accuracy on test-A versus 94.89% for a human evaluator, and GPT-4o CoT L3Score rises to 66.09 on test-A.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune on SPIQA QA data (ablation) | InstructBLIP-7B · SPIQA test-A | L3Score: 7.50 -> 43.90 | Helped |
| Fine-tune on SPIQA QA data (ablation) | LLaVA-1.5-7B · SPIQA test-A | L3Score: 13.86 -> 45.45 | Helped |
| Add full paper text (ablation) | GPT-4o · SPIQA test-A | L3Score: 64.00 -> 65.89 | Helped |
| Add full paper text (ablation) | Gemini 1.5 Pro · SPIQA test-A | L3Score: 53.49 -> 61.80 (+8.31) | Helped |
| Add CoT retrieval prompting (ablation) | GPT-4 Vision · SPIQA test-A | L3Score: 56.67 -> 63.37 (+6.70) | Helped |
| Add CoT retrieval prompting (ablation) | GPT-4o · SPIQA test-B | L3Score: 46.22 -> 46.52 | Mixed |
| Include figure and table captions (ablation) | Gemini and GPT model variants · Direct QA with figures and tables | L3Score: 2-point L3Score decrease when captions are omitted | Helped |
| Increase input image resolution (ablation) | Gemini and GPT-4 systems · SPIQA test-B | L3Score | Helped |
| Add unsupervised figure-caption training (ablation) | LLaVA-1.5-7B · SPIQA test-A | L3Score: 13.86 -> 14.43 | Helped |
| Use supervised QA fine-tuning instead of unsupervised figure-caption training (ablation) | LLaVA-1.5-7B · SPIQA test-A | L3Score: 14.43 -> 45.45 | Helped |
| Add unsupervised figure-caption pretraining before supervised QA fine-tuning (ablation) | LLaVA-1.5-7B · SPIQA test-A | L3Score: 45.45 -> 46.11 | Helped |
| Use CoT retrieval prompting (ablation) | Gemini and GPT models · SPIQA test-B | L3Score | Mixed |
Surprising findings
- Human-written and LLM-generated questions were hard to distinguish: annotators were unsure 85% of the time when asked to label the source.
- Fine-tuned InstructBLIP and LLaVA approach Gemini Pro Vision performance on L3Score despite being 7B open models.
Limitations
- Restricted to computer science papers from 19 conferences, so models may not generalize to other scientific domains.
- Test-A and training questions are LLM-generated with only partial manual filtering, and single reference answers limit free-form evaluation.
- Human baseline is a single evaluator on a 20% subset of test-A.
Experimental setup
Datasets: SPIQA train split QASA QASPER
Benchmarks: SPIQA test-A SPIQA test-B SPIQA test-C
Metrics: METEOR ROUGE-L CIDEr BERTScore F1 L3Score BLEU top-1 retrieval accuracy
Baselines: Gemini Pro Vision Gemini 1.5 Flash Gemini 1.5 Pro Claude 3 (Opus) GPT-4 Vision GPT-4o SPHINX-v2 InstructBLIP-7B LLaVA-1.5-7B XGen-MM InternLM-XC CogVLM
Builds on: QASPER QASA LLaVA 1.5 InstructBLIP LAVE
Closest papers by meaning
- MicroVQA: A Multimodal Reasoning Benchmark for Microscopy-Based Scientific Research (CVPR 2025 · similarity 0.89)
- Benchmarking Retrieval-Augmented Multimomal Generation for Document Question Answering (NeurIPS 2025 · similarity 0.89)
- cPAPERS: A Dataset of Situated and Multimodal Interactive Conversations in Scientific Papers (NeurIPS 2024 · similarity 0.87)
- UPME: An Unsupervised Peer Review Framework for Multimodal Large Language Model Evaluation (CVPR 2025 · similarity 0.87)
- Multimodal Hypothetical Summary for Retrieval-based Multi-image Question Answering (AAAI 2025 · similarity 0.87)
- InfoChartQA: A Benchmark for Multimodal Question Answering on Infographic Charts (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.