Research map / Vision-language and multimodal / Multimodal LLMs

SeePhys: Does Seeing Help Thinking? – Benchmarking Vision-Based Physics Reasoning

NeurIPS 2025 · Vision-language and multimodal › Multimodal LLMs · approach: Chain-of-thought multimodal reasoning · 2 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces SeePhys, a 2,000-question multimodal physics benchmark (75% vision-essential, 21 diagram types, middle school to PhD levels) and evaluates 28 LLMs/MLLMs under text+vision, caption, text-only and vision-only settings.

Problem

Existing physics benchmarks either lack visual inputs or treat diagrams as supplementary, so it is unclear how much current LLMs/MLLMs actually rely on and can use visual information for physics reasoning.

Approach

A physics benchmark where most questions are vision-essential (diagram info required to solve), with four input settings (TV, TC, TO, VO) and a composite LLM plus SymPy judging pipeline to quantify visual dependence across knowledge levels and diagram types.

Key results

• Gemini-2.5-Pro achieves the best model total accuracy of 54.9% on SeePhys, while human experts reach 86.5%.
• o4-mini reaches 51.9% total accuracy; most open-source MLLMs score under 25%.
• Removing diagrams (Vision Only) lowers o4-mini on Vision-Essential from 46.5% (TV) to 35.7%, while o3-mini drops 78.4% relative, indicating heavy reliance on OCR-like text reading.

Measured effects reported in the paper

Surprising findings

• Even vision-optional problems improve with diagrams for some models (e.g., Claude-3.7-Sonnet Delta2 of 56.1%), suggesting diagrams aid abstraction even when not essential.
• Accuracy does not decrease monotonically with knowledge level; senior undergraduate and advanced Olympiad problems are harder than PhD qualifying exam problems for many models.

Limitations

• Evaluation relies on LLM-based judging rather than fully automated verification, which is costly and uncertain.
• Outcome-only evaluation; no process reward or step-level assessment.
• Questions drawn from theoretical physics sources with little coverage of real-world engineering or sensory problems.

Experimental setup

Datasets: SeePhys

Benchmarks: SeePhys

Metrics: accuracy

Baselines: o4-mini o3-mini o1 Gemini-2.5-Pro Claude-3.7-Sonnet Doubao-1.5-pro GPT-4.1 GPT-4o DeepSeek-R1 Qwen2.5-VL-72B

Builds on: chain-of-thought MathVerse

Closest papers by meaning

• VGR: Visual Grounded Reasoning (ICLR 2026 · similarity 0.89)
• UGPhysics: A Comprehensive Benchmark for Undergraduate Physics Reasoning with Large Language Models (ICML 2025 · similarity 0.89)
• ChartMuseum: Testing Visual Reasoning Capabilities of Large Vision-Language Models (NeurIPS 2025 · similarity 0.88)
• VisuLogic: A Benchmark for Evaluating Visual Reasoning in Multi-modal Large Language Models (ICLR 2026 · similarity 0.88)
• MARVEL: Multidimensional Abstraction and Reasoning through Visual Evaluation and Learning (NeurIPS 2024 · similarity 0.88)
• Beyond Static Vision: Scene Dynamic Field Unlocks Intuitive Physics Understanding in Multi-modal Large Language Models (ICLR 2026 · similarity 0.88)