Research map / Multi-agent systems and games / LLM agents
FML-bench: A Controlled Study of AI Research Agent Strategies from the Perspective of Search Dynamics
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces FML-bench, an 18-task ML research benchmark with shared execution infrastructure and 12 process metrics, used to compare six AI research agents and propose an adaptive greedy-to-multi-branch search agent (AdaptiveSearch).
Problem
Which AI research agent strategy choices (search topology) drive performance is unclear because agents ship with their own execution infrastructure and evaluations report only final scores.
Approach
Standardize execution infrastructure (code editor, runner, val/test separation) across agents so only the search strategy varies, add twelve process-level behavioral metrics, and propose AdaptiveSearch that starts greedy and switches to multi-branch exploration upon detecting validation stagnation.
Key results
- TAS v2 reaches mean normalized test improvement 0.193 (58.5% win-rate) and Autoresearch 0.192 (57.4%), well above AIRA and TAS v1 (0.132 each).
- AdaptiveSearch achieves mean improvement 0.208 and 58.6% win-rate, above TAS v2 and Autoresearch on the 18 tasks.
- AUC-over-steps has pooled Spearman rho=+0.784 (p<0.001) with final test improvement; token cost (rho=-0.005) and wall-clock time show no significant association.
Surprising findings
- The simplest greedy hill-climber nearly matches the most complex best-first tree search agent in aggregate.
- The theoretically principled UCT-MCTS agent (AIRA) ranks last, and token cost is decoupled from performance.
Limitations
- Native code editors and auxiliary subsystems were removed from agents, which may underestimate their original performance.
- Six agents do not cover the full space of agent strategies.
- Fixed 100-step budget may affect relative rankings.
Experimental setup
Datasets: ColoredMNIST OfficeHome Mini-ImageNet CIFAR-100 CIFAR-10 splitMNIST IHDP MNIST SVHN COMPAS Adult TOFU
Benchmarks: FML-bench
Metrics: normalized test improvement pairwise win-rate AUC-over-steps first-improvement step Spearman correlation exploration reach effective dimension token cost wall-clock time
Baselines: Autoresearch TAS v2 TAS v1 AIDE AIRA OpenEvolve
Builds on: AIDE AI Scientist v2 AlphaEvolve Autoresearch AIRA
Closest papers by meaning
- AI Research Agents for Machine Learning: Search, Exploration, and Generalization in MLE-bench (NeurIPS 2025 · similarity 0.91)
- MARS: Modular Agent with Reflective Search for Automated AI Research (ICML 2026 · similarity 0.91)
- InnovatorBench: Evaluating Agents’ Ability to Conduct Innovative AI Research (ICLR 2026 · similarity 0.90)
- AstaBench: Rigorous Benchmarking of AI Agents with a Scientific Research Suite (ICLR 2026 · similarity 0.90)
- Agentick: A Unified Benchmark for General Sequential Decision-Making Agents (NeurIPS 2026 · similarity 0.89)
- SciResearchBench: Benchmarking AI Agents on Complex Scientific Literature Discovery (NeurIPS 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.