Research map / Multi-agent systems and games / LLM agents
MLAgentBench: Evaluating Language Agents on Machine Learning Experimentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces MLAgentBench, a 13-task benchmark for language agents doing ML experimentation, plus a ReAct-style agent with reflection/plan/fact-check, evaluated across several LMs with Claude v3 Opus best at 37.5% success.
Problem
Whether LM-driven agents can carry out end-to-end machine learning experimentation (designing, running, analyzing, and iterating on experiments) effectively, which has no standardized evaluation.
Approach
A task-agnostic environment where each ML task is defined by a description, starter files, and an automatic evaluator; a ReAct-based agent augmented with Reflection, Research Plan and Status, and Fact Check fields, plus compound LM-backed actions (Understand File, Edit Script). Success is defined as a 10% improvement over the starter-code baseline.
Key results
- Claude v3 Opus agent achieves the best average success rate of 37.5% across 13 tasks (8 runs each), reaching 100% on house-price.
- GPT-4-turbo agent is the most token-efficient, spending 51.0% fewer tokens than the average agent, with average success rate 26%.
- Own agent outperforms AutoGPT and LangChain adaptations on both GPT-4-turbo (26.0 vs 2.9 and 1.0) and Claude v3 Opus (37.5 vs 13.5 and 33.7).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Claude v3 Opus as the agent LM instead of other LMs | ReAct-style ML experimentation agent · MLAgentBench 13 tasks | average success rate (%): 37.5 vs 26.0 for next best (Claude v2.1, GPT-4-turbo) | Helped |
| Adding the structured thinking format (Reflection, Research Plan and Status, Fact Check, Thought) before actions, versus plain ReAct-style agent via LangChain | GPT-4-turbo agent · MLAgentBench 13 tasks | average success rate (%): 26.0 vs 1.0 (LangChain) | Helped |
| Adding the structured thinking format versus LangChain ReAct baseline | Claude v3 Opus agent · MLAgentBench 13 tasks | average success rate (%): 37.5 vs 33.7 (LangChain) | Mixed |
| Use the structured agent instead of AutoGPT | Claude v3 Opus agent · MLAgentBench 13 tasks | average success rate (%): 37.5 vs 13.5 | Helped |
| Running agents for longer (more steps) instead of stopping earlier | most LM agents · CIFAR-10 intermediate-step evaluation | average performance metric over time | Hurt |
| Fact Check entry in agent response (ablation) | ReAct-style agent · CIFAR-10 qualitative | hallucinated improvement claims | Helped |
Surprising findings
- Success ranges from 100% on the older house-price Kaggle task to 0% on recent Kaggle challenges and BabyLM, suggesting possible dependence on data recency.
- Running longer generally degrades performance for most models except Claude v3 Opus.
Limitations
- Success rate is low and varies widely across tasks; agents struggle with long-horizon planning and hallucinated progress.
- Averaging improvements is dominated by outliers (e.g., identify-contrails), and the improvement metric is undefined for tasks without baselines.
- Only 8 runs per agent-task pair, so estimates are noisy.
Experimental setup
Datasets: CIFAR-10 imdb ogbn-arxiv house-price spaceship-titanic parkinsons-disease fathomnet feedback identify-contrails CLRS BabyLM llama-inference
Benchmarks: MLAgentBench
Metrics: success rate average percentage improvement over baseline wall clock time input/output tokens
Baselines: AutoGPT LangChain zero-shot-react-description ReAct
Builds on: ReAct Reflexion AutoGPT LangChain
Closest papers by meaning
- MLRC-Bench: Can Language Agents Solve Machine Learning Research Challenges? (NeurIPS 2025 · similarity 0.89)
- MLR-Bench: Evaluating AI Agents on Open-Ended Machine Learning Research (NeurIPS 2025 · similarity 0.89)
- InnovatorBench: Evaluating Agents’ Ability to Conduct Innovative AI Research (ICLR 2026 · similarity 0.88)
- MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering (ICLR 2025 · similarity 0.88)
- MLE-Dojo: Interactive Environments for Empowering LLM Agents in Machine Learning Engineering (NeurIPS 2025 · similarity 0.87)
- From Reproduction to Replication: Evaluating Research Agents with Progressive Code Masking (ICLR 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.