Research map / Large language models / Agents, code and math
AutoMMLab: Automatically Generating Deployable Models from Language Instructions for Computer Vision Tasks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AutoMMLab, an LLM-driven platform that turns natural-language CV model requests into deployable models, with the LAMP benchmark and two fine-tuned LLaMA-2-7B agents (RU-LLaMA for request parsing, HPO-LLaMA for hyperparameter optimization).
Problem
Non-expert users cannot easily produce task-specific computer vision models; existing AutoML systems automate only single steps and still require expert input, and there is no benchmark for end-to-end language-instructed model production.
Approach
An end-to-end request-to-model pipeline (request understanding, data selection, model selection, HPO-driven training, deployment) in which a LoRA-fine-tuned LLaMA-2-7B parses requests into JSON configs and a second LoRA-fine-tuned LLaMA-7B proposes hyperparameters from logged trial results, trained on GPT-4-generated requests and 8,000 training runs.
Key results
- RU-LLaMA (7B) reaches 98.20 key-level and 86.25 req-level accuracy on request understanding, above GPT-4 (96.53 / 80.00).
- HPO-LLaMA with 3 rounds reaches 0.983 on classification and 0.738 on keypoint detection, above Bayesian and GPT-4 baselines.
- AutoMMLab with RU-LLaMA and HPO-LLaMA scores 112/160 end-to-end, versus 87 for GPT-4 and 84 for GPT-3.5-turbo.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune LLaMA-2 for request understanding (RU-LLaMA) instead of using GPT-4 | Request understanding · LAMP | Key-level total accuracy: 96.53 -> 98.20 | Helped |
| Fine-tune LLaMA-2 for request understanding (RU-LLaMA) instead of using GPT-4 | Request understanding · LAMP | Req-level accuracy: 80.00 -> 86.25 | Helped |
| Use HPO-LLaMA instead of the strongest listed baseline for each task | Hyperparameter optimization · LAMP image classification | accuracy: 0.964$\pm$0.056 -> 0.975$\pm$0.028 | Helped |
| Use HPO-LLaMA instead of the strongest listed baseline for each task | Hyperparameter optimization · LAMP object detection | mAP: 0.434$\pm$0.147 -> 0.435$\pm$0.148 | Helped |
| Use HPO-LLaMA instead of the strongest listed baseline for each task | Hyperparameter optimization · LAMP semantic segmentation | mIoU: 0.852$\pm$0.044 -> 0.854$\pm$0.042 | Helped |
| Use HPO-LLaMA instead of the strongest listed baseline for each task | Hyperparameter optimization · LAMP keypoint detection | mAP: 0.719$\pm$0.079 -> 0.728$\pm$0.051 | Helped |
| Increase HPO-LLaMA optimization from 1 to 3 rounds (ablation) | HPO-LLaMA · LAMP image classification | accuracy: 0.975$\pm$0.028 -> 0.983$\pm$0.020 | Helped |
| Increase HPO-LLaMA optimization from 1 to 3 rounds (ablation) | HPO-LLaMA · LAMP object detection | mAP: 0.435$\pm$0.148 -> 0.440$\pm$0.150 | Helped |
| Increase HPO-LLaMA optimization from 1 to 3 rounds (ablation) | HPO-LLaMA · LAMP semantic segmentation | mIoU: 0.854$\pm$0.042 -> 0.856$\pm$0.043 | Helped |
| Increase HPO-LLaMA optimization from 1 to 3 rounds (ablation) | HPO-LLaMA · LAMP keypoint detection | mAP: 0.728$\pm$0.051 -> 0.738$\pm$0.053 | Helped |
| Use RU-LLaMA and HPO-LLaMA instead of GPT-4 for both stages | AutoMMLab · LAMP end-to-end evaluation | Total score: 87 -> 112 | Helped |
Surprising findings
- GPT-3.5-turbo and GPT-4 fail to propose useful hyperparameters for keypoint detection despite good results on other tasks.
- LLaMA2-7B-Chat without fine-tuning gets 0 req-level accuracy and scores 0 end-to-end.
Limitations
- Evaluation covers only four CV tasks and 80 test requests.
- Random-sampling and LLM HPO baselines use small trial budgets (1 to 10 rounds).
- Request understanding and HPO are fine-tuned only on GPT-4-generated and author-curated data.
Experimental setup
Datasets: ImageNet1k COCO Cityscapes AP10K Pascal VOC 2012
Benchmarks: LAMP
Metrics: key-level accuracy req-level accuracy top-1 accuracy mAP mIoU OKS-based mAP end-to-end score (0-2 per task, max 160)
Baselines: Random sampling BayesianRF BayesianGP GPT-3.5-turbo GPT-4 PaLM2 LLaMA2-7B-Chat
Builds on: LoRA MMDeploy OpenMMLab toolboxes
Closest papers by meaning
- AutoML-Agent: A Multi-Agent LLM Framework for Full-Pipeline AutoML (ICML 2025 · similarity 0.88)
- MLZero: A Multi-Agent System for End-to-end Machine Learning Automation (NeurIPS 2025 · similarity 0.87)
- Language Models as Black-Box Optimizers for Vision-Language Models (CVPR 2024 · similarity 0.85)
- IPO: Interpretable Prompt Optimization for Vision-Language Models (NeurIPS 2024 · similarity 0.85)
- CRAFT: Customizing LLMs by Creating and Retrieving from Specialized Toolsets (ICLR 2024 · similarity 0.84)
- Mordal: Automated Pretrained Model Selection for Vision Language Models (ICLR 2026 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.