Research map / Large language models / Agents, code and math
Scaling Physical Reasoning with the PHYSICS Dataset
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces PHYSICS, a bilingual physics QA dataset of 16,568 problems extracted from over 100 textbooks, plus a Rule+Model evaluation framework (rule verifier and a fine-tuned xVerify judge) and SFT experiments on Qwen/Llama/Mistral models.
Problem
Physics is under-represented in LLM reasoning research: existing physics datasets lack quality control, scale, difficulty breadth, and train/test splits, and math-oriented answer evaluators mishandle units and numerical simplification in physics answers.
Approach
Builds a large, difficulty- and subject-diverse physics dataset from textbooks via OCR, GPT-4o extraction and expert review, with translated bilingual copies and a train/test split; pairs rule-based verification (math-verify plus unit conversion rules) with a model judge fine-tuned (physics-xVerify) so that the evaluator is accurate for physics answers.
Key results
- o3 (high) achieves the best hybrid accuracy of 58.90% on the PHYSICS test set, with DeepSeek-R1 at 55.30%, while most other models remain below 50%.
- physics-xVerify reaches 95.92% accuracy on the human-annotated evaluation set versus 83.34% for xVerify and 58.58% for GPT-4o with the rule verifier.
- SFT on 4,000 QwQ-32B traces raises Qwen2.5-7B-Instruct from 22.30% to 32.85% on PHYSICS and Qwen2.5-14B-Instruct from 27.65% to 40.20%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Combine Rule-Verifier with physics-xVerify instead of using Rule-Verifier alone | PHYSICS evaluation · Human-annotated evaluation test set | accuracy (%): 38.62 -> 95.92 | Helped |
| Fine-tune Qwen2.5-7B-Instruct on PHYSICS training data | Qwen2.5-7B-Instruct · PHYSICS | accuracy: 22.30 -> 32.85 | Helped |
| Fine-tune Qwen2.5-7B-Instruct on PHYSICS training data | Qwen2.5-7B-Instruct · GPQA (Physics) | accuracy: 35.68 -> 42.73 | Helped |
| Fine-tune Qwen2.5-7B-Instruct on PHYSICS training data | Qwen2.5-7B-Instruct · OlympiadBench (Physics) | accuracy: 23.36 -> 31.90 | Helped |
| Fine-tune Qwen2.5-7B-Instruct on PHYSICS training data | Qwen2.5-7B-Instruct · UGPhysics | accuracy: 19.67 -> 27.74 | Helped |
| Fine-tune Qwen2.5-7B-Instruct on PHYSICS training data | Qwen2.5-7B-Instruct · MATH-500 | accuracy: 76.00 -> 81.60 | Helped |
| Fine-tune Qwen2.5-7B-Instruct on PHYSICS training data | Qwen2.5-7B-Instruct · AIME-2025 | accuracy: 0.00 -> 13.33 | Helped |
| Train Qwen2.5-7B-base with PHYSICS data and reasoning paths rather than use the base model | Qwen2.5-7B-base · PHYSICS | accuracy: 7.86 -> 32.67 | Helped |
| Train Qwen2.5-14B-base with PHYSICS data and reasoning paths rather than use the base model | Qwen2.5-14B-base · PHYSICS | accuracy: 14.53 -> 35.13 | Helped |
| Train Qwen2.5-14B-base with PHYSICS data and reasoning paths rather than use the base model | Qwen2.5-14B-base · GPQA (Physics) | accuracy: 19.71 -> 52.28 | Helped |
| Train Qwen2.5-14B-base with PHYSICS data and reasoning paths rather than use the base model | Qwen2.5-14B-base · OlympiadBench (Physics) | accuracy: 12.82 -> 35.62 | Helped |
| Train Qwen2.5-14B-base with PHYSICS data and reasoning paths rather than use the base model | Qwen2.5-14B-base · UGPhysics | accuracy: 9.08 -> 37.88 | Helped |
Surprising findings
- Models score lower on high school Olympiad physics problems than on undergraduate non-physics-major problems.
- Rule-only verification underestimates accuracy by more than 20 points relative to hybrid verification across model classes.
Limitations
- Only 4,000 training samples with detailed reasoning paths are used; the rest of the training set lacks full reasoning traces.
- Judge errors persist on multi-answer, numerical simplification, and multiple-choice unit-conversion cases.
- Evaluation uses a single fixed inference configuration and a 5-gram leakage check that detects only limited contamination.
Experimental setup
Datasets: PHYSICS training split GPQA-Physics MATH-500
Benchmarks: PHYSICS GPQA-Physics OlympiadBench UGPhysics MATH-500 AIME-2025 GPQA-Diamond MMLU-Pro
Metrics: accuracy rule accuracy hybrid accuracy evaluation time
Baselines: o3 (high) DeepSeek-R1 Gemini 2.5 Pro GPT-4.1 Omni-Math-Rule Omni-Judge GPT-4o judge xVerify SimpleRL-Qwen2.5-7B General-Reasoner-Qwen2.5-7B Absolute-Zero-7B
Builds on: xVerify math-verify SimpleRL General-Reasoner Absolute Zero
Closest papers by meaning
- UGPhysics: A Comprehensive Benchmark for Undergraduate Physics Reasoning with Large Language Models (ICML 2025 · similarity 0.91)
- PhysElite: How Far Are LLMs from Solving Olympiad-Level Physics Problems? (NeurIPS 2026 · similarity 0.88)
- SeePhys: Does Seeing Help Thinking? – Benchmarking Vision-Based Physics Reasoning (NeurIPS 2025 · similarity 0.87)
- PHYBench: Holistic Evaluation of Physical Perception and Reasoning in Large Language Models (NeurIPS 2025 · similarity 0.87)
- HARDMath2: A Benchmark for Applied Mathematics Built by Students as Part of a Graduate Class (NeurIPS 2025 · similarity 0.85)
- ContPhy: Continuum Physical Concept Learning and Reasoning from Videos (ICML 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.