Research map / Multi-agent systems and games / LLM agents

MLE-Smith: Scaling MLE Tasks with Automated Multi-agent Pipeline

ICLR 2026 · Multi-agent systems and games › LLM agents · approach: Agentic benchmarks and orchestration · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes MLE-Smith, a fully automated multi-agent pipeline (Brainstormer, Designer, Refactor) that turns raw Kaggle datasets into verified competition-style MLE tasks via generate-verify-execute, producing 606 tasks and showing Elo rankings correlate strongly with human-designed MLE-Dojo tasks.

Problem

Existing MLE benchmarks and environments rely on static, manually curated competitions that are costly to build and do not scale, limiting the diversity and quantity of MLE tasks for evaluating and training agents.

Approach

A three-agent generate-verify-execute pipeline that brainstorms task formulations from raw datasets, designs and refactors them into a unified MLE task format, and checks them with deterministic assertions, LLM reviews, and execution-based validation with a ReAct MLE agent.

Key results

• Generated 606 fully verified tasks from 224 Kaggle datasets at an average cost of $0.78 per task and about 420 seconds preparation time per task.
• Elo rankings on the 50 generated tasks correlate strongly with the human-designed MLE-Dojo set (Pearson r=0.982, Spearman rho=0.952, CCC=0.958).
• Gemini-2.5-Pro is the top model on both the Dojo and Smith sets, and the GPT-4o family ranks lowest, consistent across real and generated tasks.

Surprising findings

• Designer succeeds on the first attempt in over 99% of cases, while Refactor more often needs retries and more steps.

Limitations

• Evaluation uses only 50 generated tasks and 8 LLMs, so the correlation estimate rests on a small sample.
• Pipeline depends on a single backbone (GPT-5) for all agents; compatibility with other LLMs is claimed but not shown.
• Execution-based validation time excluded from reported cost and varies widely with dataset and hardware.

Experimental setup

Datasets: 224 Kaggle datasets (300 sampled for generation) MLE-Dojo evaluation tasks

Benchmarks: MLE-Dojo Dojo set (50 tasks) MLE-Smith Smith set (50 generated tasks) Combined set (100 tasks)

Metrics: Elo rating Pearson r Spearman rho Kendall tau_b Lin's CCC Bland-Altman LoA Cronbach alpha ICC(2,1) Top-k overlap

Baselines: MLE-Dojo MLE-Bench MLGym DS-Bench MLAgentBench

Builds on: MLE-Dojo MLE-Bench ReAct

Closest papers by meaning

• MLE-Dojo: Interactive Environments for Empowering LLM Agents in Machine Learning Engineering (NeurIPS 2025 · similarity 0.92)
• MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering (ICLR 2025 · similarity 0.90)
• MLZero: A Multi-Agent System for End-to-end Machine Learning Automation (NeurIPS 2025 · similarity 0.90)
• SWE-smith: Scaling Data for Software Engineering Agents (NeurIPS 2025 · similarity 0.89)
• ML-Agent: Reinforcing LLM Agents for Autonomous Machine Learning Engineering (ICML 2026 · similarity 0.88)
• MLE-STAR: Machine Learning Engineering Agent via Search and Targeted Refinement (NeurIPS 2025 · similarity 0.88)