Research map / Vision-language and multimodal / Multimodal LLMs

Can MLLMs Reason in Multimodality? EMMA: An Enhanced MultiModal ReAsoning Benchmark

ICML 2025 · Vision-language and multimodal › Multimodal LLMs · approach: Chain-of-thought multimodal reasoning · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces EMMA, a benchmark of 2,788 multimodal reasoning questions across math, physics, chemistry, and coding, and shows that nine SoTA MLLMs, including o1, fall far short of human experts, with test-time scaling and CoT giving only limited gains.

Problem

Existing multimodal benchmarks either test surface-level visual perception or allow shortcuts through text-only reasoning, so they fail to measure whether MLLMs can reason jointly over text and images.

Approach

Filter candidate questions by captioning images with GPT-4o and discarding any question answered correctly by text-only LLMs with captions in most trials, then add newly built expert-verified questions in physics, chemistry, and coding that require visual simulation, with fine-grained skill labels and short-answer or multiple-choice formats so no MLLM judge is needed.

Key results

• o1 achieves 45.75% on EMMA-mini, the best single model, about 32% below human experts (77.75%).
• Gemini 2.0 Flash Thinking with majority voting at N=16 reaches 50.75% on EMMA-mini, versus 43.5% at N=1.
• CoT improves closed-source models (e.g., Claude 3.5 Sonnet +2.15 on EMMA) but reduces accuracy for top open-source models such as Qwen2-VL-72B (-6.35 on EMMA).

Measured effects reported in the paper

Surprising findings

• Textual CoT hurts open-source models and hurts visually heavy tasks such as 2D transformation.
• Self-reward modeling with the same model underperforms plain majority voting in best-of-N selection.

Limitations

• Physics section is small (156 questions) and chemistry is mostly organic-chemistry arrow-pushing questions.
• Rate limits prevented full-benchmark evaluation of o1, which was evaluated only on EMMA-mini.
• Filtering relies on GPT-4o captions and specific LLMs, so filtering results may depend on these choices.

Experimental setup

Datasets: MathVista Math-Vision RAVEN OlympiadBench EXAMS-V MMMU SMiCRM CharXiv matplotlib example gallery

Benchmarks: EMMA EMMA-mini

Metrics: accuracy

Baselines: Claude 3.5 Sonnet Gemini 2.0 Flash GPT-4o Qwen2-VL-72B-Instruct LLaVA-OneVision-72B InternVL2-Llama3-76B InternVL2.5-78B Gemini 2.0 Flash Thinking o1

Builds on: MMMU-Pro MathVista Math-Vision CharXiv RAVEN

Closest papers by meaning

• LENS: Multi-level Evaluation of Multimodal Reasoning with Large Language Models (ICLR 2026 · similarity 0.92)
• MMR-Life: Piecing Together Real-life Scenes for Multimodal Multi-image Reasoning (ICLR 2026 · similarity 0.92)
• Compose and Fuse: Revisiting the Foundational Bottlenecks in Multimodal Reasoning (ICLR 2026 · similarity 0.91)
• Unveiling Fine-Grained Visual Traces: Evaluating MultiModal Interleaved Reasoning Chains in Multimodal STEM Tasks (NeurIPS 2026 · similarity 0.90)
• RPTS: Tree-Structured Reasoning Process Scoring for Faithful Multimodal Evaluation (AAAI 2026 · similarity 0.90)
• AStar: Boosting Multimodal Reasoning with Automated Structured Thinking (AAAI 2026 · similarity 0.90)