Research map / Multi-agent systems and games / LLM agents
Gravity-Bench-v1: A Benchmark on Gravitational Physics Discovery for Agents
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Gravity-Bench-v1, an environment-based benchmark where LLM agents must plan observations under a budget and analyze simulated binary-star data (including out-of-distribution modified gravity and drag) to answer physics discovery questions.
Problem
Existing benchmarks rarely test whether AI agents can autonomously run the scientific discovery loop of collecting data under uncertainty, planning observations within a budget, and reasoning to quantitative answers, especially in novel physics.
Approach
A Rebound-based two-body gravity simulator exposed as a partially observable environment with an observe tool under a fixed observation budget, including out-of-distribution physics (drag, modified gravity exponent) and expert reference solutions used to set task-specific error thresholds.
Key results
- Under a 100-observation budget, o4-mini-high scores 49.4% ± 2.6%, while Claude 3.5 Sonnet scores 21.5% and GPT-4o 15.5%.
- With full table access, o4-mini-high reaches 73.9% ± 2.4%, versus 39.5% for Claude 3.5 Sonnet and 36.1% for GPT-4o.
- Models stop early and use fewer than half the available observations for budgets above 30; GPT-4o averages only 12 observations under the 100 budget.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Provide full observation data instead of limiting agents to 100 observations | Baseline AI agents on Gravity-Bench-v1 · Gravity-Bench-v1 | task score: o4-mini-high-2025-04-16: 49.4% ± 2.6% -> 73.9% ± 2.4%; Claude 3.5 Sonnet: 21.5% ± 2.5% -> 39.5% ± 3.2%; Claude 3.5 Haiku | Helped |
| Use o4-mini-high-2025-04-16 rather than the next-best tested model | Baseline agents with a 100-observation budget · Gravity-Bench-v1, budget-obs-100 | task score: 49.4% ± 2.6% vs. 21.5% ± 2.5% | Helped |
| Use o4-mini-high-2025-04-16 rather than the next-best tested model | Baseline agents with full observation access · Gravity-Bench-v1, full-obs | task score: 73.9% ± 2.4% vs. 39.5% ± 3.2% | Helped |
| Use adaptive strategic sampling instead of uniform sampling | Expert solution for periastron distance · Highly elliptical single-orbit system | percent error: 93% error -> 2% accuracy | Helped |
| Iteratively refine observations around the suspected peak-velocity interval | Claude 3.5 Sonnet · Maximum velocity of a star | percent error: 2% | Helped |
| Misplan observations by failing to track peak-velocity times | Claude 3.5 Sonnet · Maximum velocity of a star | percent error: 45% | Hurt |
| Use uniform sampling rather than adaptive sampling | Expert solution for maximum velocity · Maximum velocity of a star | percent error: 70% error | Hurt |
Surprising findings
- o4-mini-high consistently solves modified-gravity tasks with full data, suggesting generalization to rarely-seen physics, though budgeted runs fail on these OOD tasks.
- Models frequently assume unjustified stellar masses (e.g., 1.0), and such assumptions correlate with incorrect answers (GPT-4o: 33% of incorrect vs 9% of correct solutions).
Limitations
- Narrow scope: only two-body Newtonian-like gravity in a 2D plane with error-free observations.
- Task thresholds are set from expert-ref-100 performance gaps and are somewhat ad hoc (5% to 70%).
- Only OpenAI and Anthropic models tested; Claude 4 models were not included.
Experimental setup
Datasets: Rebound-simulated binary star trajectories (16 two-body simulations, 206 task-simulation pairs)
Benchmarks: Gravity-Bench-v1
Metrics: percent of tasks solved within task-specific error threshold total cost total run time mean observations used
Baselines: expert-ref-100 uniform sampling claude-3-5-sonnet-20241022 claude-3-5-haiku-20241022 gpt-4o-2024-11-20 gpt-4o-mini-2024-07-18
Builds on: ReAct
Closest papers by meaning
- PhysGym: Benchmarking LLMs in Interactive Physics Discovery with Controlled Priors (NeurIPS 2025 · similarity 0.88)
- DiscoveryWorld: A Virtual Environment for Developing and Evaluating Automated Scientific Discovery Agents (NeurIPS 2024 · similarity 0.86)
- UltraHorizon: Benchmarking LLM-Agent Capabilities in Ultra Long-Horizon Scenarios (ICML 2026 · similarity 0.86)
- AstaBench: Rigorous Benchmarking of AI Agents with a Scientific Research Suite (ICLR 2026 · similarity 0.85)
- Agentic AI Scientists Are Not Built For Autonomous Scientific Discovery (NeurIPS 2026 · similarity 0.83)
- SciResearchBench: Benchmarking AI Agents on Complex Scientific Literature Discovery (NeurIPS 2026 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.