AI Paper Atlas

Research map / Large language models / Agents, code and math

FIRE-Bench: Evaluating AI Agents on the Rediscovery of Scientific Insights

Zhen Wang, Fan Bai, Zhongyan Luo, Jinyan Su, Kaiser Sun, Xinle Yu, Jieyuan Liu, Kun Zhou, Claire Cardie, Mark Dredze, Zhiting Hu, Eric P. Xing

ICML 2026 · Large language models › Agents, code and math · approach: Long-horizon LLM agents

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
upgrade OpenHands backbone from o4-mini to gpt-5 (ablation)OpenHands · FIRE-Bench (30 core tasks, claim-level)F1: 31.9 -> 37.9Improved
upgrade OpenHands backbone from o4-mini to gpt-5 (ablation)OpenHands · FIRE-Bench (30 core tasks), total API costtotal cost (USD): 8.90 -> 10.74Hurt
use Claude Code (Sonnet-4) instead of Codex (gpt-5-medium)coding agent · FIRE-Bench (30 core tasks, claim-level)F1: 41.9 -> 46.7Improved
use Codex instead of Claude Codecoding agent · FIRE-Bench (30 core tasks), total API costtotal cost (USD): 12.67 -> 2.21Improved
task requires control-based counterfactual design instead of a direct pipelineall four agents · FIRE-Bench: LLM Racial Bias in Medicine taskF1: Lost in the Middle best 91.7; Racial Bias all <= 34.2Hurt
post-cutoff vs pre-cutoff publication of the task (Hard tasks) (ablation)OpenHands o4-mini and gpt-5 · FIRE-Bench contamination analysis (30 core tasks)F1: 15.4 -> 24.8 (o4-mini), 22.6 -> 31.0 (gpt-5)No clear effect
post-cutoff vs pre-cutoff publication of the task (Medium tasks) (ablation)OpenHands gpt-5 · FIRE-Bench contamination analysis (30 core tasks)F1: 44.5 -> 23.1 (gpt-5), 31.9 -> 33.6 (o4-mini)Mixed
switch from LLM-behavior tasks to cross-domain tasks (CV/VLM and NN analysis)agents · FIRE-Bench cross-domain extension (10 tasks)average F1: Average F1 below 25 for OH and ~30.2 for CodexHurt
use gpt-5 instead of o4-mini within OpenHands on cross-domain NN-analysis tasks (ablation)OpenHands · FIRE-Bench cross-domain extension (10 tasks)average F1: 23.4 -> 13.3Hurt

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy