Research map / Large language models / Agents, code and math
FIRE-Bench: Evaluating AI Agents on the Rediscovery of Scientific Insights
Zhen Wang, Fan Bai, Zhongyan Luo, Jinyan Su, Kaiser Sun, Xinle Yu, Jieyuan Liu, Kun Zhou, Claire Cardie, Mark Dredze, Zhiting Hu, Eric P. Xing
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| upgrade OpenHands backbone from o4-mini to gpt-5 (ablation) | OpenHands · FIRE-Bench (30 core tasks, claim-level) | F1: 31.9 -> 37.9 | Improved |
| upgrade OpenHands backbone from o4-mini to gpt-5 (ablation) | OpenHands · FIRE-Bench (30 core tasks), total API cost | total cost (USD): 8.90 -> 10.74 | Hurt |
| use Claude Code (Sonnet-4) instead of Codex (gpt-5-medium) | coding agent · FIRE-Bench (30 core tasks, claim-level) | F1: 41.9 -> 46.7 | Improved |
| use Codex instead of Claude Code | coding agent · FIRE-Bench (30 core tasks), total API cost | total cost (USD): 12.67 -> 2.21 | Improved |
| task requires control-based counterfactual design instead of a direct pipeline | all four agents · FIRE-Bench: LLM Racial Bias in Medicine task | F1: Lost in the Middle best 91.7; Racial Bias all <= 34.2 | Hurt |
| post-cutoff vs pre-cutoff publication of the task (Hard tasks) (ablation) | OpenHands o4-mini and gpt-5 · FIRE-Bench contamination analysis (30 core tasks) | F1: 15.4 -> 24.8 (o4-mini), 22.6 -> 31.0 (gpt-5) | No clear effect |
| post-cutoff vs pre-cutoff publication of the task (Medium tasks) (ablation) | OpenHands gpt-5 · FIRE-Bench contamination analysis (30 core tasks) | F1: 44.5 -> 23.1 (gpt-5), 31.9 -> 33.6 (o4-mini) | Mixed |
| switch from LLM-behavior tasks to cross-domain tasks (CV/VLM and NN analysis) | agents · FIRE-Bench cross-domain extension (10 tasks) | average F1: Average F1 below 25 for OH and ~30.2 for Codex | Hurt |
| use gpt-5 instead of o4-mini within OpenHands on cross-domain NN-analysis tasks (ablation) | OpenHands · FIRE-Bench cross-domain extension (10 tasks) | average F1: 23.4 -> 13.3 | Hurt |
Closest papers by meaning
- MLR-Bench: Evaluating AI Agents on Open-Ended Machine Learning Research (NeurIPS 2025 · similarity 0.92)
- SciResearchBench: Benchmarking AI Agents on Complex Scientific Literature Discovery (NeurIPS 2026 · similarity 0.91)
- AstaBench: Rigorous Benchmarking of AI Agents with a Scientific Research Suite (ICLR 2026 · similarity 0.91)
- InnovatorBench: Evaluating Agents’ Ability to Conduct Innovative AI Research (ICLR 2026 · similarity 0.91)
- Can AI Agents Synthesize Scientific Conclusions? (NeurIPS 2026 · similarity 0.90)
- ACED-Bench: Evaluating How LLM Agents Acquire and Act on Causal Evidence (NeurIPS 2026 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
