Research map / Vision-language and multimodal / Multimodal LLMs
Scientists' First Exam: Probing Cognitive Abilities of MLLM via Perception, Understanding, and Reasoning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces SFE, a bilingual 830-VQA benchmark over 66 expert-curated scientific tasks in five disciplines, organized by three cognitive levels (signal perception, attribute understanding, comparative reasoning), and evaluates 16 MLLMs, finding the best (GPT-o3) reaches only 34.08%.
Problem
Existing scientific multimodal benchmarks mainly test knowledge recall and rarely measure perception and reasoning over native scientific data formats, so MLLMs' scientific cognitive abilities are not granularly assessed.
Approach
Organize scientific VQA by a three-level cognitive taxonomy (perception, attribute understanding, comparative reasoning) built from native scientific raw data rendered as images, with expert-authored bilingual questions in MCQ, exact match, and open-question formats.
Key results
- GPT-o3 achieves the best average score of 34.08% (English) and 31.60% (Chinese) on SFE, while InternVL-3-78B reaches 26.52% (English).
- Gemini-2.5-Pro scores only 8.04% English average, attributed to exhausted token budgets from excessive thinking.
- GPT-o3 improves L3 comparative reasoning to 36.48% from GPT-4.1's 26.64% with little change in L2 (30.30% vs 30.02%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| newer model versions (e.g., GPT-o3 vs GPT-4.1) with reasoning training | L3 comparative reasoning · SFE L3 | LLM-as-a-Judge score (English): 26.64 -> 36.48 | Helped |
| newer model versions (GPT-o3 vs GPT-4.1) with reasoning training | L2 attribute understanding · SFE L2 | LLM-as-a-Judge score (English): 30.02 -> 30.30 | No clear effect |
| Gemini-2.5-Pro with excessive thinking under 1024 token cap | Gemini-2.5-Pro · SFE overall | LLM-as-a-Judge score (English average): 8.04 vs GPT-o3 34.08 | Hurt |
| sampling temperature raised from 0.0 to 0.2 (or 0.4-0.6) (ablation) | InternVL-3-78B · SFE | LLM-as-a-Judge score: 24.43 -> 26.98 at 0.2 | Helped |
| sampling temperature 0.0 to 0.6 (ablation) | GPT-4.1-2025-04-14 · SFE | LLM-as-a-Judge score: 30.88 -> 32.09 | Helped |
| closed-weight proprietary models vs open-weight models | SFE overall · SFE | LLM-as-a-Judge score: best proprietary outperforms InternVL-3-78B by 6-8% | Helped |
| larger model size (Qwen2.5-VL-72B vs 7B; InternVL-3-78B vs smaller) | open-weight MLLMs · SFE | LLM-as-a-Judge score | No clear effect |
| InternVL-2.5 to InternVL-3 (native multimodal pretraining and MPO) | L3 comparative reasoning · SFE L3 English | LLM-as-a-Judge score: about 8 points gain | Helped |
Surprising findings
- Gemini-2.5-Pro scores far below other models (8.04% English average), largely due to excessive reasoning consuming the token budget.
- Qwen2.5-VL-72B underperforms its 7B variant, suggesting overfitting and insufficient scientific data scaling with model size.
Limitations
- Benchmark scope and depth acknowledged by authors as limited and to be enhanced.
- Single run per experiment with temperature 0 and a 1024-token generation cap, which may penalize reasoning models.
- LLM-as-a-judge scoring with GPT-4o may introduce judging bias.
Experimental setup
Datasets: ERA5 MoleculeNet RCSB PDB PDBbind PubChem
Benchmarks: SFE
Metrics: LLM-as-a-judge score BERTScore F1 IoU execution success rate pass@k
Baselines: GPT-4o-2024-11-20 GPT-4.1-2025-04-14 GPT-o1-2024-12-17 Claude-3.7-Sonnet Gemini-2.5-Pro Gemini-2.5-Flash Qwen2.5-VL-72B InternVL-3-78B Llama-3.2-Vision-90B LLaVA-OneVision-72B
Builds on: MMMU ScienceQA CURIE HLE
Closest papers by meaning
- MME-SCI: A Comprehensive and Challenging Science Benchmark for Multimodal Large Language Models (AAAI 2026 · similarity 0.90)
- MDK12-Bench: A Multi-Discipline Benchmark for Evaluating Reasoning in Multimodal Large Language Models (AAAI 2026 · similarity 0.90)
- MicroVQA: A Multimodal Reasoning Benchmark for Microscopy-Based Scientific Research (CVPR 2025 · similarity 0.89)
- RBench: Graduate-level Multi-disciplinary Benchmarks for LLM & MLLM Complex Reasoning Evaluation (ICML 2025 · similarity 0.88)
- Can MLLMs Reason in Multimodality? EMMA: An Enhanced MultiModal ReAsoning Benchmark (ICML 2025 · similarity 0.87)
- LENS: Multi-level Evaluation of Multimodal Reasoning with Large Language Models (ICLR 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.