Research map / Large language models / Agents, code and math
UGPhysics: A Comprehensive Benchmark for Undergraduate Physics Reasoning with Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces UGPhysics, a bilingual (EN/ZH) 11,040-problem undergraduate physics benchmark with a model-assisted rule-based answer judge (MARJ), evaluating 31 LLMs; best is OpenAI-o1-mini at 49.8%.
Problem
Existing LLM benchmarks cover mostly high-school or competition physics with multiple-choice formats, and lack breadth and depth at the undergraduate level. Answer judging for physics is also unreliable because of constants, units, and equivalent expressions.
Approach
A large undergraduate-level physics benchmark spanning 13 subjects and 7 answer types, with a two-stage judge that applies physics-aware rule matching first and a GPT-4o judge only for rule-flagged failures, plus skill labels and leakage checks.
Key results
- OpenAI-o1-mini achieves the highest overall accuracy of 49.78% on UGPhysics, with DeepSeek-R1 at 57.16% EN overall reported in newly-added results.
- DeepSeek-R1-Distill-Llama-70B achieves 40.17% overall, while 15 of 31 evaluated LLMs score below 20%.
- MARJ achieves 98% agreement with human annotations on a 100-example subset.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| increase Qwen2.5-Math model size from 7B to 72B | Qwen2.5-Math-Instruct · UGPhysics overall | accuracy: 22.69 -> 39.52 | Helped |
| increase DeepSeek-R1-Distill-Llama model size from 8B to 70B | DeepSeek-R1-Distill-Llama · UGPhysics overall | accuracy: 13.11 -> 40.17 | Helped |
| use math-specialized training rather than general-purpose training | Qwen2.5-7B · UGPhysics overall | accuracy: 22.55 -> 22.69 | Helped |
| use math-specialized training rather than general-purpose training | Qwen2.5-72B · UGPhysics overall | accuracy: 36.22 -> 39.52 | Helped |
| increase token limit from 8192 to 16382 (ablation) | DeepSeek-R1-Distill-Qwen-7B · UGPhysics overall | accuracy: 24.64 -> 24.86 | Helped |
| increase token limit from 8192 to 16382 (ablation) | DeepSeek-R1-Distill-Qwen-32B · UGPhysics overall | accuracy: 31.93 -> 32.21 | Helped |
| increase token limit from 8192 to 16382 (ablation) | DeepSeek-R1-Distill-Llama-8B · UGPhysics overall | accuracy: 13.11 -> 14.51 | Helped |
| increase token limit from 8192 to 16382 (ablation) | DeepSeek-R1-Distill-Llama-70B · UGPhysics overall | accuracy: 40.17 -> 41.77 | Helped |
| increase token limit from 8192 to 16382 (ablation) | QwQ-32B-Preview · UGPhysics overall | accuracy: 37.34 -> 38.90 | Helped |
Surprising findings
- Math-specialized LLMs yield only marginal gains over general-purpose counterparts on physics (Qwen2.5-Math-7B-Instruct only 0.14% higher average than Qwen2.5-7B-Instruct).
- Error analysis of OpenAI-o1-mini shows flawed reasoning, knowledge deficiency, and wrong application dominate, unlike math where calculation errors are common.
Limitations
- Only text-only problems; problems with images were excluded.
- Estimation, proof, and explanation problems excluded, so open-ended physics reasoning is not covered.
- Judge relies on GPT-4o for second-stage evaluation, and human check covered only 100 examples.
Experimental setup
Datasets: The Great Compendium of Physics Problems (USTC, seven books)
Benchmarks: UGPhysics MMLU MMLU-pro OlympicArena GPQA MATH
Metrics: accuracy clip ratio n-gram leakage detection rate human agreement
Baselines: OpenAI-o1-mini GPT-4o GPT-4o-mini Qwen2.5-72B-Instruct DeepSeek-R1-Distill-Llama-70B QwQ-32B-Preview Qwen2.5-Math-72B-Instruct
Builds on: OlympiadBench GPT-4o-based evaluation Mathpix OCR
Closest papers by meaning
- Scaling Physical Reasoning with the PHYSICS Dataset (NeurIPS 2025 · similarity 0.91)
- UGMathBench: A Diverse and Dynamic Benchmark for Undergraduate-Level Mathematical Reasoning with Large Language Models (ICLR 2025 · similarity 0.91)
- PhysElite: How Far Are LLMs from Solving Olympiad-Level Physics Problems? (NeurIPS 2026 · similarity 0.90)
- SeePhys: Does Seeing Help Thinking? – Benchmarking Vision-Based Physics Reasoning (NeurIPS 2025 · similarity 0.89)
- SciBench: Evaluating College-Level Scientific Problem-Solving Abilities of Large Language Models (ICML 2024 · similarity 0.88)
- PHYBench: Holistic Evaluation of Physical Perception and Reasoning in Large Language Models (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.