Research map / Multi-agent systems and games / LLM agents
MLZero: A Multi-Agent System for End-to-end Machine Learning Automation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces MLZero, a multi-agent LLM system with a perception module and semantic/episodic memory that automates end-to-end multimodal ML from raw data, evaluated on MLE-Bench Lite and a new 25-dataset Multimodal AutoML Agent Benchmark.
Problem
Existing AutoML and LLM-agent systems need heavy manual configuration, rely on preprocessed inputs, and struggle with hallucinated code and outdated library APIs when handling raw multimodal data end to end.
Approach
A cognitive perception module turns raw multimodal files into perceptual context and selects an ML library; iterative code generation is augmented with semantic memory (condensed library tutorials retrieved by summary) and episodic memory (error summaries and fix suggestions) across nine agents.
Key results
- On the Multimodal AutoML Agent Benchmark, MLZero (default) reaches 92.0% success rate and 2.42 average rank, versus 14.7%-25.3% for baseline agents with default settings.
- On MLE-Bench Lite, MLZero achieves 86% success rate and average rank 1.43, with six gold medals, ahead of AIDE (81%).
- With an 8B Llama 3.1 backbone, MLZero still reaches 45.3% success rate and 5.14 average rank, outperforming several full-size default baselines.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use MLZero instead of competing agents | MLE-Bench Lite · MLE-Bench Lite | success rate: 86% vs AIDE (81%), MLAB (62%), and OpenHands (71%) | Helped |
| add MLZero's perception, memory, and iterative coding system | Multimodal AutoML Agent Benchmark agents · Multimodal AutoML Agent Benchmark | success rate: 92.0% vs 14.7% (Codex CLI), 25.3% (AIDE), and 13.3% (DS-Agent) | Helped |
| replace the default LLM with Llama 3.1 8B (ablation) | MLZero · Multimodal AutoML Agent Benchmark | success rate: 92.0% -> 45.3% | Hurt |
| remove access to external ML libraries (ablation) | MLZero · Multimodal AutoML Agent Benchmark | success rate: 92.0% -> 69.3% | Hurt |
| add episodic memory (ablation) | MLZero · Multimodal AutoML Agent Benchmark | success rate: 86.7% -> 92.0% | Helped |
| extend runtime to 24 hours and improve the quality preset (ablation) | MLZero · Multimodal AutoML Agent Benchmark | solution quality | Helped |
| add suggested fixes to error summaries (ablation) | MLZero episodic memory · Episodic memory ablation | success rate: 0.75 -> 1.0 | Helped |
| add error summaries and suggested fixes (ablation) | MLZero episodic memory · Episodic memory ablation | success rate: 0.83 -> 1.0 | Helped |
| remove condensation from semantic memory tutorials (ablation) | MLZero semantic memory · Semantic memory ablation | token consumption | Hurt |
| use summary-based rather than title-based retrieval (ablation) | MLZero semantic memory · Semantic memory ablation | retrieval performance | Helped |
| increase semantic-memory retrieval from zero to five documents (ablation) | MLZero semantic memory · Semantic memory ablation | performance | Helped |
| increase semantic-memory retrieval beyond five documents (ablation) | MLZero semantic memory · Semantic memory ablation | performance | No clear effect |
Surprising findings
- Removing suggested fixes from episodic memory lowered success (0.75) and worsened token efficiency, suggesting over-condensed error summaries can confuse the coder.
- Codex CLI with a reasoning model improves substantially (14.7% to 69.3% success), while its default setup gives simple, quickly executing solutions with degraded performance.
Limitations
- Many MLE-Bench Lite tasks cannot be handled by currently integrated ML libraries, requiring additional library registration.
- Error frequency comparisons are influenced by how far each system progresses, so failure-category statistics are not directly comparable.
- Baselines required manual intervention or extraction of outputs, so comparisons involve some non-end-to-end adaptations.
Experimental setup
Datasets: MLE-Bench Lite TabRepo datasets RVL-CDIP MLDoc Memotion HAM10000 Hateful Memes Kickstarter funding PetFinder Airbnb Melbourne Women's E-Commerce Clothing Reviews
Benchmarks: MLE-Bench Lite Multimodal AutoML Agent Benchmark
Metrics: success rate average rank relative time medal count token efficiency
Baselines: Codex CLI AIDE DS-Agent AutoKaggle MLAB (ResearchAgent) OpenHands (CodeActAgent)
Builds on: AIDE MLAgentBench DS-Agent AutoKaggle OpenHands
Closest papers by meaning
- AutoML-Agent: A Multi-Agent LLM Framework for Full-Pipeline AutoML (ICML 2025 · similarity 0.91)
- MLE-Smith: Scaling MLE Tasks with Automated Multi-agent Pipeline (ICLR 2026 · similarity 0.90)
- ML-Agent: Reinforcing LLM Agents for Autonomous Machine Learning Engineering (ICML 2026 · similarity 0.89)
- MLE-Dojo: Interactive Environments for Empowering LLM Agents in Machine Learning Engineering (NeurIPS 2025 · similarity 0.89)
- AutoMMLab: Automatically Generating Deployable Models from Language Instructions for Computer Vision Tasks (AAAI 2025 · similarity 0.87)
- MLR-Bench: Evaluating AI Agents on Open-Ended Machine Learning Research (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.