Research map / Large language models / Agents, code and math
OptiMUS: Scalable Optimization Modeling with (MI)LP Solvers and Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces OptiMUS, a modular multi-agent LLM system (manager, formulator, programmer, evaluator) with a connection graph that formulates and solves LP/MILP problems from natural language, plus a new NLP4LP benchmark.
Problem
Automating optimization modeling from natural-language descriptions is hard for LLMs due to ambiguous terms, long problem descriptions, large data, and unreliable code/formulations.
Approach
A manager-coordinated team of LLM agents (formulator, programmer, evaluator) that processes each constraint/objective clause independently via a connection graph linking clauses, parameters, and variables, with data separated from the problem text so prompts stay short.
Key results
- OptiMUS with GPT-4 reaches 78.8% on NL4OPT, 66.7% on ComplexOR and 72.0% on NLP4LP, versus 64.2%, 38.1% and 53.1% for Chain-of-Experts.
- Replacing GPT-4 with GPT-3.5 drops accuracy to 28.6% on NL4OPT, 9.5% on ComplexOR and 14.4% on NLP4LP.
- Removing debugging lowers accuracy to 72.3% on NL4OPT, 57.1% on ComplexOR and 58.2% on NLP4LP.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use OptiMUS instead of CoE | GPT-4 optimization modeling · NL4OPT | accuracy: 64.2% -> 78.8% | Helped |
| use OptiMUS instead of CoE | GPT-4 optimization modeling · ComplexOR | accuracy: 38.1% -> 66.7% | Helped |
| use OptiMUS instead of CoE | GPT-4 optimization modeling · NLP4LP | accuracy: 53.1% -> 72.0% | Helped |
| add programmer debugging (ablation) | OptiMUS (GPT-4) · NL4OPT | accuracy: 72.3% -> 78.7% | Helped |
| add programmer debugging (ablation) | OptiMUS (GPT-4) · ComplexOR | accuracy: 57.1% -> 66.7% | Helped |
| add programmer debugging (ablation) | OptiMUS (GPT-4) · NLP4LP | accuracy: 58.2% -> 71.6 % | Helped |
| use GPT-4 rather than GPT-3.5 for the manager (ablation) | OptiMUS · NL4OPT | accuracy: 74.9% -> 78.7% | Helped |
| use GPT-4 rather than GPT-3.5 for the manager (ablation) | OptiMUS · ComplexOR | accuracy: 52.4% -> 66.7% | Helped |
| use GPT-4 rather than GPT-3.5 for the manager (ablation) | OptiMUS · NLP4LP | accuracy: 53.7% -> 71.6 % | Helped |
| use GPT-4 instead of GPT-3.5 across agents (ablation) | OptiMUS · NL4OPT | accuracy: 28.6% -> 78.7% | Helped |
| use GPT-4 instead of GPT-3.5 across agents (ablation) | OptiMUS · ComplexOR | accuracy: 9.5% -> 66.7% | Helped |
| use GPT-4 instead of GPT-3.5 across agents (ablation) | OptiMUS · NLP4LP | accuracy: 14.4% -> 71.6 % | Helped |
Surprising findings
- Using GPT-3.5 for the manager and GPT-4 for other agents causes only a small drop on NL4OPT but a larger one on the harder datasets.
- Smaller open LLMs such as Mixtral-8x7B perform very poorly (6.6% on NL4OPT, 0% on ComplexOR) without fine-tuning.
Limitations
- Relies on closed GPT-4; smaller LLMs perform poorly out of the box.
- Failure analysis shows incorrect modeling and missing constraints are common on harder datasets.
- ComplexOR subset augmented with synthetic data; accuracy only reported, not compile/runtime error rates.
Experimental setup
Datasets: NL4Opt ComplexOR NLP4LP
Benchmarks: NL4Opt ComplexOR NLP4LP
Metrics: accuracy compilation error rate runtime error rate average prompt length
Baselines: Standard prompting Reflexion Chain-of-Experts
Builds on: Reflexion Chain-of-Experts
Closest papers by meaning
- OptiBench Meets ReSocratic: Measure and Improve LLMs for Optimization Modeling (ICLR 2025 · similarity 0.89)
- Autoformulation of Mathematical Optimization Models Using LLMs (ICML 2025 · similarity 0.88)
- OptMATH: A Scalable Bidirectional Data Synthesis Framework for Optimization Modeling (ICML 2025 · similarity 0.88)
- OptiTree: Hierarchical Thoughts Generation with Tree Search for LLM Optimization Modeling (NeurIPS 2025 · similarity 0.87)
- MM-OptBench: A Solver-Grounded Benchmark for Multimodal Optimization Modeling (NeurIPS 2026 · similarity 0.87)
- Enhancing Decision Making Through the Integration of Large Language Models and Operations Research Optimization (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.