Research map / Multi-agent systems and games / LLM agents
AutoML-Agent: A Multi-Agent LLM Framework for Full-Pipeline AutoML
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AutoML-Agent, a multi-agent LLM framework that automates full-pipeline AutoML (data retrieval to deployment) using retrieval-augmented planning, prompting-based training-free execution, and multi-stage verification.
Problem
Existing LLM-based AutoML systems cover only single pipeline steps or specific task types, and search methods often require costly actual training; a task-agnostic, full-pipeline AutoML system is needed that is accessible to non-experts and efficient.
Approach
A manager agent devises multiple end-to-end plans using retrieved external knowledge (web search, arXiv), decomposes each plan into role-specific sub-tasks for data and model agents that execute via prompting (simulated, not run), and applies request, execution, and implementation verification with feedback-driven plan revision before code generation.
Key results
- AutoML-Agent achieves an average success rate of 87.1% in the constraint-aware setting, outperforming baselines.
- Compared with SELA (10 rollouts) on six datasets, AutoML-Agent runs about 8x faster with an average score of 0.612 vs 0.599.
- On average a single run takes about 525 seconds and costs 0.30 USD with GPT-4o to produce a deployable model.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use AutoML-Agent instead of the baselines | Full-pipeline AutoML · Constraint-aware setting, average across tasks | success rate | Helped |
| Use AutoML-Agent instead of SELA (10 rollouts) | Full-pipeline AutoML · Six datasets | average score: 0.599 -> 0.612 | Helped |
| Use prompting-based search instead of SELA's training-based search | AutoML-Agent · Six datasets | search time: 8x faster | Helped |
| Use retrieval-augmented planning with plan decomposition and verification rather than retrieval-augmented planning alone (ablation) | AutoML-Agent · Downstream tasks | comprehensive score | Helped |
| Add plan decomposition to retrieval-augmented planning (ablation) | AutoML-Agent · Downstream tasks | downstream performance | Helped |
| Add code verification to RAP with plan decomposition (ablation) | AutoML-Agent · Downstream tasks | runnable-model success | Helped |
| Vary the number of plans (ablation) | AutoML-Agent · Constraint-free setting | success rate | No clear effect |
| Vary the number of plans (ablation) | AutoML-Agent · Constraint-free setting | downstream task performance | Mixed |
| Inject noisy examples before or after insight extraction (ablation) | AutoML-Agent · Simulated noise scenarios | final model performance | No clear effect |
| Vary system prompt phrasing while keeping agent roles clearly defined (ablation) | AutoML-Agent · Prompt sensitivity study | framework performance | No clear effect |
Surprising findings
- Small open models (LLaMA-2-7B, Qwen1.5-14B, gemma-7B-it, WizardLM-70B, CodeQwen1.5-7B) and even vanilla GPT-3.5/GPT-4o failed to generate executable code for complex full-pipeline tasks, per the appendix.
- Injecting noisy retrieved insights sometimes improved performance rather than degrading it.
Limitations
- Relies on closed-source GPT-4o; noticeable quality gap with weaker backbones such as GPT-3.5.
- Lack of skeleton code for new tasks increases risk of code hallucination.
- Does not yet cover reinforcement learning or recommendation pipelines.
Experimental setup
Datasets: Butterfly Image Classification Shopee-IET Ecommerce Text Textual Entailment Banana Quality Software Defects Crab Age Crop Price Smoker Status Higher Education Students Performance Weather Electricity
Benchmarks: Butterfly Image Classification Shopee-IET Ecommerce Text Textual Entailment Banana Quality Software Defects Crab Age Crop Price Smoker Status Higher Education Students Performance Weather Electricity
Metrics: success rate normalized performance score comprehensive score accuracy F1 RMSLE Rand index search time monetary cost
Baselines: Human Models AutoGluon DS-Agent GPT-3.5 GPT-4 SELA
Builds on: DS-Agent EvolInstruct LoRA HuggingGPT
Closest papers by meaning
- MLZero: A Multi-Agent System for End-to-end Machine Learning Automation (NeurIPS 2025 · similarity 0.91)
- ML-Agent: Reinforcing LLM Agents for Autonomous Machine Learning Engineering (ICML 2026 · similarity 0.89)
- AutoTool: Efficient Tool Selection for Large Language Model Agents (AAAI 2026 · similarity 0.89)
- AutoMMLab: Automatically Generating Deployable Models from Language Instructions for Computer Vision Tasks (AAAI 2025 · similarity 0.88)
- MARS: Modular Agent with Reflective Search for Automated AI Research (ICML 2026 · similarity 0.88)
- MLE-Dojo: Interactive Environments for Empowering LLM Agents in Machine Learning Engineering (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.