Research map / Large language models / Agents, code and math

PhysGym: Benchmarking LLMs in Interactive Physics Discovery with Controlled Priors

NeurIPS 2025 · Large language models › Agents, code and math · approach: Long-horizon LLM agents · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces PhysGym, a benchmark of 97 PHYBench-derived physics equation-discovery environments where LLM agents run experiments under a budget, with four controllable prior-knowledge levels, and evaluates six LLMs by symbolic-equivalence success rate.

Problem

Existing benchmarks for LLM scientific reasoning lack fine-grained control over how much prior knowledge an agent receives, so it is unclear whether models discover physical laws through interaction or recall memorized patterns.

Approach

An interactive equation-discovery simulator in which the textual context, variable descriptions and variable names are progressively masked across four prior levels, with budgeted experiments and automatic symbolic/LLM equivalence checking of the final hypothesis.

Key results

• Success rates fall from about 62-74% at Level 1 to 12-32% at Level 4 for the top models (e.g., Gemini-2.5-pro 74.23% to 31.96%; o4-mini 62.89% to 27.84%).
• Some problems are solved only at lower prior levels (e.g., Environment 716 solved only at L3 and L4 by Gemini-flash and o4-mini).
• Claude-3.7-Sonnet reaches only 22.68% at L1 and 12.37% at L4, and gains little from richer priors.

Measured effects reported in the paper

Surprising findings

• Some problems are solved at lower prior levels but fail when full context is provided, showing prior knowledge can constrain exploration.
• Claude-3.7-Sonnet's fit quality worsens with more prior knowledge, suggesting it anchors on priors.

Limitations

• Static problem set of 97 manually constructed environments drawn from PHYBench.
• Task difficulty is approximated by number of variables and equation length.
• Two-stage evaluation (SymPy then LLM judge) introduces some subjectivity.

Experimental setup

Datasets: PHYBench

Benchmarks: PhysGym

Metrics: success rate R^2 MSE Kendall's tau MAPE experiments per solved task unique hypotheses

Baselines: Gemini-2.5-flash OpenAI o4-mini Claude-3.7-Sonnet Gemini-2.5-pro gpt-oss-20b Qwen3-32B

Builds on: PHYBench Schmidhuber artificial curiosity

Closest papers by meaning

• LLM and Simulation as Bilevel Optimizers: A New Paradigm to Advance Physical Scientific Discovery (ICML 2024 · similarity 0.90)
• DiscoveryWorld: A Virtual Environment for Developing and Evaluating Automated Scientific Discovery Agents (NeurIPS 2024 · similarity 0.89)
• SciAgentGym: Benchmarking Multi-Step Scientific Tool-Use in LLM Agents (ICML 2026 · similarity 0.89)
• Gravity-Bench-v1: A Benchmark on Gravitational Physics Discovery for Agents (ICML 2025 · similarity 0.88)
• ScienceBoard: Evaluating Multimodal Autonomous Agents in Realistic Scientific Workflows (ICLR 2026 · similarity 0.88)
• PHYBench: Holistic Evaluation of Physical Perception and Reasoning in Large Language Models (NeurIPS 2025 · similarity 0.88)