Research map / Large language models / Agents, code and math
HiPhO: How Far Are (M)LLMs from Humans in the Latest High School Physics Olympiad Benchmark?
Fangchen Yu, Haiyuan Wan, Qianjia Cheng, Yuchen Zhang, Jiacheng Chen, Fujun Han, Yulun Wu, Junchi Yao, Ruilizhen Hu, Ning Ding, Yu Cheng, Tao Chen and 5 more
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| move from text-only to text plus data figure problems | Gemini-2.5-Pro · HiPhO modality types | mean normalized score (%): 86 -> 67 | Hurt |
| replace GPT-5's variable-figure score with an open-source MLLM | Qwen2.5-VL-72B-Instruct vs GPT-5 · HiPhO Text+Variable Figure | mean normalized score (%): 75 -> 29 | Hurt |
| switch from other fields to optics problems | Gemini-2.5-Pro · HiPhO physics fields | mean normalized score (%): 89 -> 55 | Hurt |
| add extended reasoning (thinking mode) to Claude-4-Sonnet | Claude-4-Sonnet · HiPhO (13 Olympiads) | gold medals: 2 -> 8 | Improved |
| use reasoning model DeepSeek-R1 instead of chat model DeepSeek-V3 | DeepSeek-V3 · HiPhO (13 Olympiads) | gold medals: 1 -> 8 | Improved |
| use high reasoning effort for o4-mini | o4-mini · HiPhO (13 Olympiads) | gold medals: 7 -> 6 | Improved |
| use Gemini-2.5-Flash as grader instead of GPT-4o | grading pipeline · HiPhO step-level scoring | agreement with human experts: under 1-point differences | Improved |
| compare best model to top human at IPhO-2025 | Gemini-2.5-Pro · IPhO-2025 | exam score: 29.2 -> 22.7 | Hurt |
| use open-source Kimi-K2-Instruct on PanMechanics-2025 versus top human | Kimi-K2-Instruct · PanMechanics-2025 | exam score: 62.0 -> 65.9 | Improved |
Closest papers by meaning
- PhysElite: How Far Are LLMs from Solving Olympiad-Level Physics Problems? (NeurIPS 2026 · similarity 0.84)
- Scaling Physical Reasoning with the PHYSICS Dataset (NeurIPS 2025 · similarity 0.84)
- UGPhysics: A Comprehensive Benchmark for Undergraduate Physics Reasoning with Large Language Models (ICML 2025 · similarity 0.84)
- LiveCodeBench Pro: How Do Olympiad Medalists Judge LLMs in Competitive Programming? (NeurIPS 2025 · similarity 0.82)
- LiveBench: A Challenging, Contamination-Limited LLM Benchmark (ICLR 2025 · similarity 0.81)
- Leveraging Online Olympiad-Level Math Problems for LLMs Training and Contamination-Resistant Evaluation (ICML 2025 · similarity 0.81)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
