Research map / Vision-language and multimodal / Multimodal LLMs

MME-SCI: A Comprehensive and Challenging Science Benchmark for Multimodal Large Language Models

AAAI 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Benchmarks and analysis of MLLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces MME-SCI, a 1,019-item multilingual (5 languages), multimodal (text-only, image-only, image-text) science benchmark across math, physics, chemistry, and biology with 63 fine-grained knowledge points, and evaluates 20 MLLMs on it.

Problem

Existing multimodal science benchmarks are saturated and lack multilingual coverage, comprehensive modality coverage (image-only and text-only modes), and fine-grained knowledge point annotation for diagnosing MLLM weaknesses.

Approach

A manually curated, multilingual science benchmark (Chinese high-school level, translated into English, French, Spanish, Japanese) with three input modes (text-only, image-only screenshots, image-text) and fine-grained knowledge point labels, filtered by human solvers to keep only challenging questions.

Key results

• o4-mini achieved 52.11%, 24.73%, 36.57%, and 29.80% accuracy in math, physics, chemistry, and biology under the Image-only mode.
• Qwen2.5VL-72B, the top open-source model, reached only 19.43% average accuracy in Image-only mode, versus 41.32% for Doubao-Seed-1.6.
• Reasoning-tuned Kimi-VL-A3B-Thinking achieved a 72.88% gain over its Instruct version in Image-only scenarios.

Surprising findings

• Open-source models lose accuracy on screenshot-based image-only inputs, while closed-source Doubao-1.5-thinking-vision-pro gains 5.20%.
• Qwen2.5-Omni-7B shows a 4.71% drop on image-only questions compared with its vision-language base Qwen2.5-VL-7B.

Limitations

• Only 1,019 questions, so per-knowledge-point sample sizes are small.
• Questions are sourced from Chinese high-school mock exams, limiting scope to high-school level science.
• Evaluation relies on an LLM-as-a-judge paradigm whose judge model is not specified.

Experimental setup

Datasets: MME-SCI

Benchmarks: MME-SCI

Metrics: accuracy

Baselines: MMMU AI2D GAOKAO-Bench MathVerse MATH-Vision EMMA GeoSense PhyX VisioMath

Builds on: MMMU GAOKAO-Bench MathVerse MATH-Vision EMMA

Closest papers by meaning

• MDK12-Bench: A Multi-Discipline Benchmark for Evaluating Reasoning in Multimodal Large Language Models (AAAI 2026 · similarity 0.93)
• SEED-Bench: Benchmarking Multimodal Large Language Models (CVPR 2024 · similarity 0.92)
• MME-CoT: Benchmarking Chain-of-Thought in Large Multimodal Models for Reasoning Quality, Robustness, and Efficiency (ICML 2025 · similarity 0.91)
• HSSBench: Benchmarking Humanities and Social Sciences Ability for Multimodal Large Language Models (ICLR 2026 · similarity 0.91)
• BMMR: A Large-Scale Bilingual Multimodal Multi-Discipline Reasoning Dataset (NeurIPS 2025 · similarity 0.91)
• Scientists' First Exam: Probing Cognitive Abilities of MLLM via Perception, Understanding, and Reasoning (NeurIPS 2025 · similarity 0.90)