Research map / Large language models / Agents, code and math
PhysElite: How Far Are LLMs from Solving Olympiad-Level Physics Problems?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces PhysElite, a bilingual multimodal benchmark of 11,586 Olympiad-level physics problems with diagrams and step-by-step solutions, and evaluates 18 MLLMs/LLMs with answer and process scores, finding the best model reaches 33.7% answer accuracy.
Problem
Existing physics benchmarks are either too easy (K-12/undergraduate) or small and unimodal, and they lack step-level annotations, so current models' true Olympiad-level physics ability is unclear.
Approach
A large-scale bilingual (Chinese-English) multimodal Olympiad physics benchmark sourced from contestant training materials, with diagrams, human-verified step-by-step derivations, and a two-metric (answer accuracy and step-level process score) evaluation using an LLM-judge ensemble.
Key results
- Best model Grok-4.2 reaches 33.7% answer accuracy on PhysElite, while the human baseline reaches 48.5%.
- Claude-Opus-4.6 reaches 28.1% answer accuracy and 49.6 process score; strongest open-source model Qwen3-VL-235B-A22B reaches 20.0%.
- Process diagrams improve answer and process scores for all six tested models, while textual solution hints give only marginal changes.
Surprising findings
- Claude-Opus-4.6 in default mode outperforms five of six reasoning models, suggesting base capability matters more than extended thinking.
- Medium-difficulty problems are harder than Hard ones for many frontier models (U-shape), attributed to post-training on Olympiad-style data.
Limitations
- Partial-credit scoring introduces annotation ambiguity in borderline cases.
- Possible minor overlap of training data with commercial model training sets.
- Problems sourced from Chinese contestant materials, which may limit generality.
Experimental setup
Datasets: PhysElite
Benchmarks: PhysElite
Metrics: answer accuracy process score first-error index
Baselines: Grok-4.2 Claude-Opus-4.6 o3-mini GPT-5.2 Gemini-3-Pro Kimi-K2-Thinking Claude-Sonnet-4.5 Gemini-2.5 Qwen-VL-Max GPT-4o Qwen3-VL-235B-A22B DeepSeek-V3
Builds on: OlympiadBench PHYBench PhysReason HiPhO
Closest papers by meaning
- UGPhysics: A Comprehensive Benchmark for Undergraduate Physics Reasoning with Large Language Models (ICML 2025 · similarity 0.90)
- PHYBench: Holistic Evaluation of Physical Perception and Reasoning in Large Language Models (NeurIPS 2025 · similarity 0.89)
- Scaling Physical Reasoning with the PHYSICS Dataset (NeurIPS 2025 · similarity 0.88)
- CURIE: Evaluating LLMs on Multitask Scientific Long-Context Understanding and Reasoning (ICLR 2025 · similarity 0.87)
- SciBench: Evaluating College-Level Scientific Problem-Solving Abilities of Large Language Models (ICML 2024 · similarity 0.87)
- PhysGym: Benchmarking LLMs in Interactive Physics Discovery with Controlled Priors (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.