Research map / Multi-agent systems and games / LLM agents
MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces MLE-bench, a benchmark of 75 offline Kaggle ML engineering competitions with human-leaderboard-based medal grading, and evaluates frontier LLMs (o1-preview, GPT-4o, Claude 3.5 Sonnet, Llama 3.1 405B) with AIDE, MLAB, and OpenHands scaffolds.
Problem
Few benchmarks measure autonomous end-to-end machine learning engineering by AI agents with comparison to human-level performance; existing ones are small or cover narrower tasks.
Approach
Port 75 manually vetted Kaggle competitions into an offline environment with re-created train/test splits and local grading, and grade agent submissions against Kaggle Private leaderboards using Kaggle's medal thresholds, so agents are directly comparable to human competitors.
Key results
- o1-preview with AIDE achieves at least a Kaggle bronze medal in 16.9% of competitions (pass@1), roughly doubling to 34.1% at pass@8.
- GPT-4o (AIDE) reaches pass@6 of 17.0%, comparable to o1-preview pass@1.
- Scaling GPT-4o runtime from 24 to 100 hours raises medal rate from 8.7% to 11.8%; extra GPU or CPU-only hardware changes results little (8.7-10.2%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use AIDE instead of MLAB or OpenHands with GPT-4o | GPT-4o agent scaffold · MLE-bench 75 competitions | Any medal (%): 0.8 ± 0.5 and 4.4 ± 1.4 -> 8.7 ± 0.5 | Helped |
| Use o1-preview instead of GPT-4o with AIDE | AIDE · MLE-bench 75 competitions | Any medal (%): 8.7 ± 0.5 -> 16.9 ± 1.1 | Helped |
| Increase attempts from pass@1 to pass@8 | o1-preview + AIDE · MLE-bench 75 competitions | Any medal (%): 16.9% -> 34.1% | Helped |
| Increase attempts from pass@1 to pass@6 | GPT-4o + AIDE · MLE-bench 75 competitions | Any medal (%): 8.7% -> 17.0% | Helped |
| Change hardware between CPU-only, one A10 GPU, and two A10 GPUs | GPT-4o + AIDE · MLE-bench 75 competitions | Any medal (%): 9.1 \pm 1.0; 8.7 \pm 0.5; 10.2 \pm 2.0 | No clear effect |
| Obfuscate competition descriptions | GPT-4o + AIDE · MLE-bench 75 competitions | Any medal (%): $8.5 \pm 0.6$ -> $8.4 \pm 1.0$ | No clear effect |
| Increase runtime from 24 hours to 100 hours per competition | GPT-4o + AIDE · MLE-bench 75 competitions | Any medal (%): 8.7% -> 11.8% | Helped |
Surprising findings
- Giving agents an extra GPU made no measurable difference, and logs showed no attempts to use the second GPU.
- No systematic contamination effect: GPT-4o's familiarity with competition pages showed no correlation with performance, and obfuscated descriptions gave similar scores (8.5% vs 8.4%).
Limitations
- Contamination from public Kaggle material cannot be fully ruled out for future models, and high-level strategy reuse is hard to detect.
- Re-created train/test splits and re-implemented grading mean scores are only approximately comparable to Kaggle leaderboards.
- Resource intensive: a single full run needs about 1800 GPU hours and many tokens.
Experimental setup
Datasets: Kaggle competitions (75 test, 7 development) Meta Kaggle
Benchmarks: MLE-bench
Metrics: any medal rate bronze/silver/gold rate valid submission rate above median rate pass@k
Baselines: MLAB OpenHands Claude 3.5 Sonnet Llama 3.1 405B GPT-4o
Builds on: AIDE MLAgentBench OpenHands CodeActAgent pass@k
Closest papers by meaning
- AI Research Agents for Machine Learning: Search, Exploration, and Generalization in MLE-bench (NeurIPS 2025 · similarity 0.93)
- MLR-Bench: Evaluating AI Agents on Open-Ended Machine Learning Research (NeurIPS 2025 · similarity 0.92)
- MLRC-Bench: Can Language Agents Solve Machine Learning Research Challenges? (NeurIPS 2025 · similarity 0.91)
- MLE-Smith: Scaling MLE Tasks with Automated Multi-agent Pipeline (ICLR 2026 · similarity 0.90)
- CoMind: Towards Community-Driven Agents for Machine Learning Engineering (ICLR 2026 · similarity 0.89)
- MLE-STAR: Machine Learning Engineering Agent via Search and Targeted Refinement (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.