Research map / Large language models / Agents, code and math
PHYBench: Holistic Evaluation of Physical Perception and Reasoning in Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces PHYBench, a 500-problem original, human-curated physics reasoning benchmark, and the EED Score, a tree-edit-distance partial-credit metric for symbolic answers.
Problem
Existing LLM reasoning benchmarks suffer from oversimplified tasks, data contamination, and flawed items, and binary scoring fails to capture partial correctness in symbolic answers.
Approach
Physics problems written from scratch by students and vetted through multi-round review, with a symbolic-expression answer format scored by an Expression Edit Distance (tree edit distance over SymPy trees, with discounted subtree insertion/deletion) that gives partial credit.
Key results
- Gemini 2.5 Pro is the best model at 36.9% accuracy and EED Score 49.5, versus human experts at 61.9% accuracy and 70.4 EED.
- EED Score improves sample efficiency by 204% on average, so 500 problems with EED match about 1500 problems with binary accuracy.
- DeepSeek-R1 averages 10636 output tokens on PHYBench, far more than MATH-500, GPQA, OlympiadBench, or AIME 2024.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use EED Score instead of binary accuracy for evaluation | PHYBench evaluation · PHYBench | sample efficiency: 204% | Helped |
| Evaluate on PHYBench rather than established reasoning benchmarks | DeepSeek-R1 · PHYBench compared with MATH-500, GPQA, OlympiadBench, and AIME 2024 | accuracy: 97.3 -> 25.0 | Hurt |
| Evaluate on PHYBench rather than established reasoning benchmarks | DeepSeek-R1 · PHYBench compared with MATH-500, GPQA, OlympiadBench, and AIME 2024 | average output tokens: 1857 -> 10636 | Mixed |
| Increase pass@$k sampling at test time | Models evaluated on PHYBench · PHYBench | pass@$k accuracy | Helped |
| Provide a correct partial solution trace | DeepSeek-V3 · PHYBench | accuracy: $66$ -> $93$ | Helped |
| Provide a correct partial solution trace | Gemini 2.5 Pro · PHYBench | accuracy: $97$ -> $100$ | Helped |
| Provide a correct partial solution trace | DeepSeek-R1 · PHYBench | accuracy: $97$ -> $98$ | Helped |
Surprising findings
- Perturbed-trace experiments show DeepSeek-R1 and Gemini 2.5 Pro rely on compensatory symbolic checks rather than genuine semantic understanding, with DeepSeek-R1 dropping to 37% under one perturbation.
- Over 90% of model errors occur in robust reasoning rather than physical perception, and most of those are semantic reasoning errors.
Limitations
- Focus on Olympiad-level problems and uneven topic distribution limits generalization to research-level reasoning.
- EED scores only the final expression and does not assess intermediate reasoning steps.
- EED treats all nodes equally and does not check physical plausibility such as dimensional correctness.
Experimental setup
Datasets: PHYBench
Benchmarks: PHYBench MATH-500 AIME 2024 OlympiadBench GPQA
Metrics: accuracy EED Score pass@k majority voting accuracy sample efficiency
Baselines: MATH-500 AIME 2024 OlympiadBench GPQA SuperGPQA MMLU
Builds on: Zhang-Shasha tree edit distance SymPy latex2sympy_extended
Closest papers by meaning
- PhysElite: How Far Are LLMs from Solving Olympiad-Level Physics Problems? (NeurIPS 2026 · similarity 0.89)
- metabench - A Sparse Benchmark of Reasoning and Knowledge in Large Language Models (ICLR 2025 · similarity 0.89)
- SciBench: Evaluating College-Level Scientific Problem-Solving Abilities of Large Language Models (ICML 2024 · similarity 0.89)
- DyVal: Dynamic Evaluation of Large Language Models for Reasoning Tasks (ICLR 2024 · similarity 0.88)
- UGPhysics: A Comprehensive Benchmark for Undergraduate Physics Reasoning with Large Language Models (ICML 2025 · similarity 0.88)
- MDK12-Bench: A Multi-Discipline Benchmark for Evaluating Reasoning in Multimodal Large Language Models (AAAI 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.