Research map / Multi-agent systems and games / LLM agents

MLE-Dojo: Interactive Environments for Empowering LLM Agents in Machine Learning Engineering

NeurIPS 2025 · Multi-agent systems and games › LLM agents · approach: Agentic benchmarks and orchestration · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces MLE-Dojo, a Gym-style interactive environment built on 200+ Kaggle MLE tasks (150 train, 50 eval) for training and evaluating LLM agents with iterative feedback, and benchmarks eight frontier LLMs using AUP, HumanRank and Elo.

Problem

Existing MLE agent benchmarks rely on static datasets or single-attempt evaluations and lack interactive, executable environments that support iterative experimentation and training (SFT/RL) for LLM agents.

Approach

A Gym-style, Docker-isolated environment with standardized Kaggle task format, five native actions (request_info, validate_code, execute_code, get_history, reset), and a HumanRank reward (relative position on the human leaderboard) that gives continuous feedback across heterogeneous metrics.

Key results

• Gemini-2.5-Pro achieves the highest MLE-Lite Elo (1257) and H-Rank 61.95% on MLE-Lite among evaluated models.
• o3-mini reaches 56.48% H-Rank on MLE-Lite, while gpt-4o-mini reaches 21.21%.
• CV tasks are the hardest: none of the tasks have average HumanRank above 60, and more than half fall below 30.

Surprising findings

• DeepSeek-r1 achieves competitive performance with significantly fewer tokens, though it has relatively high execution and validation failure rates.
• Conservative models (gpt-4o, gpt-4o-mini) execute code only about 20% of the time and rely heavily on validation, yet still face submission validity issues.

Limitations

• Evaluation relies on API-based frontier models; no open-model training or RL experiments reported.
• Substantial computational and storage demands of executing full MLE pipelines.
• Licensing and privacy constraints of Kaggle-derived data; only a first release of 200 tasks.

Experimental setup

Datasets: MLE-Bench DSBench Kaggle competitions (scraped)

Benchmarks: MLE-Dojo

Metrics: AUP HumanRank score (H-Rank %) Elo

Baselines: AutoKaggle MLAgentBench DSBench DSEval MLE-Bench MLGym

Builds on: MLE-Bench DSBench MLGym MLAgentBench

Closest papers by meaning

• ML-Agent: Reinforcing LLM Agents for Autonomous Machine Learning Engineering (ICML 2026 · similarity 0.93)
• MLE-Smith: Scaling MLE Tasks with Automated Multi-agent Pipeline (ICLR 2026 · similarity 0.92)
• FT-Dojo: Towards Autonomous LLM Fine-Tuning with Language Agents (ICML 2026 · similarity 0.91)
• MLE-STAR: Machine Learning Engineering Agent via Search and Targeted Refinement (NeurIPS 2025 · similarity 0.89)
• MLZero: A Multi-Agent System for End-to-end Machine Learning Automation (NeurIPS 2025 · similarity 0.89)
• MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering (ICLR 2025 · similarity 0.88)