Research map / Multi-agent systems and games / LLM agents

BioAgent Bench: An AI Agent Evaluation Suite for Bioinformatics

ICML 2026 · Multi-agent systems and games › LLM agents · approach: Agentic benchmarks and orchestration · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces BioAgent Bench, ten end-to-end bioinformatics tasks with LLM-graded pipeline completion and robustness perturbations (corrupted inputs, decoys, prompt bloat), evaluating frontier and open-weight models in three agent harnesses.

Problem

Existing evaluations collapse bioinformatics workflows into QA or code generation, so they cannot measure end-to-end pipeline completion, robustness, or step-level reliability of AI agents on multi-step bioinformatics tasks.

Approach

A curated suite of end-to-end bioinformatics pipeline tasks with concrete output artifacts, scored by an LLM grader on pipeline step completion, plus controlled perturbations (corrupted inputs, decoy files, prompt bloat) and multi-trial stability metrics (Jaccard, Pearson).

Key results

• Claude Opus 4.5 reached 100% completion in Codex CLI; Gemini 3 Pro, GPT-5.2 and Sonnet 4.5 exceeded 90%, while best open-weight model GLM-4.7 reached 82.5%.
• Prompt bloat reduced completion by 28% relative to clean runs; corrupted inputs were flagged in 7/10 tasks and decoys were used erroneously in 2/10 tasks.
• Across four GPT-5.2 trials per task, mean Jaccard overlap was 0.43 and mean Pearson correlation was 0.73, indicating substantial run-to-run variability.

Surprising findings

• Pipeline completion can substantially overestimate reliability: agents often build correct pipelines yet fail to detect corrupted inputs or avoid decoy files.
• Some models (e.g., Gemini 3 Pro) produced weaker plans than frontier models yet still completed pipelines at high rates.

Limitations

• LLM grader subjectivity and bias; grader may reward plausible-looking traces.
• Resource caps (<4h runtime, 48GB RAM) limit realism, favoring small organisms.
• Robustness analysis uses a single trial per task and condition, and only GPT-5.2 was analyzed for stability.

Experimental setup

Datasets: CEPH_1463 Complete Genomics family data GIAB NA12878 exome Candida parapsilosis RNA-seq E. coli experimental evolution reads Cuatro Cienegas metagenomics reads skeletal muscle single-cell RNA-seq

Benchmarks: BioAgent Bench

Metrics: pipeline completion rate steps completed final result reached results match F1 score Jaccard index Pearson correlation plan quality score (1-5)

Baselines: GPT-5.1 Gemini 3 Pro GLM-4.7 Kimi K2 Thinking Devstral 2512 MiniMax M2.1 Qwen3-Coder

Builds on: SWE-bench

Closest papers by meaning

• BioXArena: Benchmarking LLM Agents on Multi-Modal Biomedical Machine Learning Tasks (NeurIPS 2026 · similarity 0.88)
• AstaBench: Rigorous Benchmarking of AI Agents with a Scientific Research Suite (ICLR 2026 · similarity 0.87)
• BioDiscoveryAgent: An AI Agent for Designing Genetic Perturbation Experiments (ICLR 2025 · similarity 0.85)
• MLR-Bench: Evaluating AI Agents on Open-Ended Machine Learning Research (NeurIPS 2025 · similarity 0.85)
• Simple Agents Outperform Experts in Biomedical Imaging Workflow Optimization (CVPR 2026 · similarity 0.85)
• AutoScientists: Self-Organizing Agent Teams for Long-Running Scientific Experimentation (NeurIPS 2026 · similarity 0.85)