Research map / Large language models / Agents, code and math
BMMR: A Large-Scale Bilingual Multimodal Multi-Discipline Reasoning Dataset
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces BMMR, a bilingual (English/Chinese) multimodal multi-discipline college-level reasoning dataset with a 20k eval split, an 89k train split, and a process-based verifier (BMMR-Verifier) for step-level scoring of reasoning paths.
Problem
Existing multimodal benchmarks lack subject diversity, reasoning depth, and language coverage, and the open-source community lacks multimodal multi-discipline training data with curated reasoning paths; outcome-only exact-match scoring also misses flawed reasoning.
Approach
Combine a large UNESCO-taxonomy-based bilingual multimodal dataset (eval + train, with reasoning paths) with a process reward-style verifier trained on step-level +/- labels from 32 rollouts per question, enabling fine-grained reasoning-step evaluation and fine-tuning of open models.
Key results
- Gemini-2.5-Pro achieves 50.15 overall on BMMR-Eval while o3 gets 38.06, leaving substantial headroom.
- Fine-tuning on BMMR-Train yields a 19.07% overall improvement for BMMR-InternVL2.5-78B.
- BMMR-Verifier agrees with GPT-4o at 91.67% response-level and 89.21% step-level, versus human agreement of 95.00% and 93.71%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune Qwen2.5-VL-3B-Instruct on BMMR-Train | Qwen2.5-VL-3B-Instruct · BMMR-Eval ICTs | performance improvement: 72.28% improvement | Helped |
| Fine-tune InternVL2.5-78B on BMMR-Train | InternVL2.5-78B · BMMR-Eval Health | performance improvement: 43.34% improvement | Helped |
| Fine-tune open-source models on BMMR-Train | Open-source models (3B–78B) · BMMR-Eval | overall performance: 19.07% improvement | Helped |
| Increase Qwen2.5-VL model size from 3B to 7B, 32B, and 72B | Qwen2.5-VL · BMMR-Eval | overall performance: 13.57, 26.07, 34.09, and 38.22 | Helped |
| Use chain-of-thought prompting instead of non-CoT prompting (ablation) | Evaluated models, especially reasoning-tuned InternVL2.5-MPO and InternVL3 · BMMR-Eval | overall performance | Helped |
Surprising findings
- Reasoning models (LRMs) do not consistently outperform LMMs across disciplines; o3 scores 63.89 in ICTs but 27.78 in Health.
- Chain-of-thought prompting substantially boosts performance over non-CoT prompting for most models, with some reasoning-tuned models showing the largest drops without it.
Limitations
- Covers only English and Chinese questions and does not include all UNESCO subjects.
- Data sourced from print and digital materials introduces quality and safety uncertainty, mitigated by GPT-4o filtering and human checks.
- Several top models evaluated on a 5.4k TestMini subset due to cost constraints.
Experimental setup
Datasets: BMMR-Train BMMR-Eval MMMU MathVista MathVerse
Benchmarks: BMMR-Eval
Metrics: accuracy step-level agreement response-level agreement
Baselines: MMMU MathVista MathVerse MMMU-Pro CMMMU ScienceQA R-Bench GPT-4o
Builds on: MMMU process reward models Llama Factory MS-Swift
Closest papers by meaning
- MME-SCI: A Comprehensive and Challenging Science Benchmark for Multimodal Large Language Models (AAAI 2026 · similarity 0.91)
- RBench: Graduate-level Multi-disciplinary Benchmarks for LLM & MLLM Complex Reasoning Evaluation (ICML 2025 · similarity 0.91)
- MME-CoT: Benchmarking Chain-of-Thought in Large Multimodal Models for Reasoning Quality, Robustness, and Efficiency (ICML 2025 · similarity 0.91)
- MDK12-Bench: A Multi-Discipline Benchmark for Evaluating Reasoning in Multimodal Large Language Models (AAAI 2026 · similarity 0.90)
- MMR-Life: Piecing Together Real-life Scenes for Multimodal Multi-image Reasoning (ICLR 2026 · similarity 0.90)
- LENS: Multi-level Evaluation of Multimodal Reasoning with Large Language Models (ICLR 2026 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.