Research map / Large language models / Agents, code and math
HeurekaBench: A Benchmarking Framework for AI Co-scientist
Siba Smarak Panigrahi, Jovana Videnović, Maria Brbic
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add end-critic (ablation) | Biomni (GPT-OSS-120B) · sc-HeurekaBench OEQs | correctness: 2.08 -> 2.40 | Improved |
| add end-critic (ablation) | Biomni (Qwen3-235B-thinking) · sc-HeurekaBench OEQs | correctness: 1.85 -> 1.73 | Hurt |
| add plan-critic (ablation) | Biomni (GPT-OSS-120B) · sc-HeurekaBench OEQs | correctness: 2.08 -> 1.92 | Hurt |
| add plan-critic (ablation) | Biomni (Qwen3-235B-thinking) · sc-HeurekaBench OEQs | correctness: 1.85 -> 1.56 | Hurt |
| add retriever module (ablation) | Biomni (GPT-OSS-120B) · sc-HeurekaBench-TU | correctness: 1.56 -> 2.15 | Improved |
| add retriever module (ablation) | Biomni (Qwen3-235B-thinking) · sc-HeurekaBench-TU | correctness: 1.80 -> 1.92 | Improved |
| use Claude-4-Sonnet instead of GPT-OSS-120B as planner | Biomni · sc-HeurekaBench OEQs | correctness: 2.08 -> 2.58 | Improved |
| use thinking variant of Qwen3-235B as planner (ablation) | Biomni · sc-HeurekaBench OEQs | correctness: 1.57 -> 1.85 | Improved |
| scale Qwen3 planner from 32B to 235B (ablation) | Biomni · sc-HeurekaBench OEQs | correctness: 1.47 -> 1.57 | Improved |
| use GPT-OSS-120B instead of Qwen3-235B as planner | Biomni · sc-HeurekaBench OEQs | correctness: 1.57 -> 2.08 | Improved |
| use Biomni instead of CellVoyager | Claude-4-Sonnet agent · sc-HeurekaBench-Lite MCQs | accuracy: 27.78 -> 50.00 | Improved |
Closest papers by meaning
- AstaBench: Rigorous Benchmarking of AI Agents with a Scientific Research Suite (ICLR 2026 · similarity 0.91)
- InnovatorBench: Evaluating Agents’ Ability to Conduct Innovative AI Research (ICLR 2026 · similarity 0.89)
- SciResearchBench: Benchmarking AI Agents on Complex Scientific Literature Discovery (NeurIPS 2026 · similarity 0.89)
- SciAgentGym: Benchmarking Multi-Step Scientific Tool-Use in LLM Agents (ICML 2026 · similarity 0.89)
- MLR-Bench: Evaluating AI Agents on Open-Ended Machine Learning Research (NeurIPS 2025 · similarity 0.88)
- Fisher-R1: Training LLM Agents for Reliable Hypothesis Testing (NeurIPS 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
