Research map / Large language models / Agents, code and math
MM-OptBench: A Solver-Grounded Benchmark for Multimodal Optimization Modeling
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces multimodal optimization modeling and MM-OptBench, a 780-instance solver-verified benchmark where MLLMs must produce a formulation and solver code from text plus visual artifacts, finding best pass@1 near 52%.
Problem
Existing optimization-modeling benchmarks are text-only, while real optimization specifications often carry instance data in tables, graphs, maps, and schedules. Whether MLLMs can build correct formulations and solver code from such text-plus-visual specifications is untested.
Approach
A solver-grounded generation framework that derives model-facing text and visuals, hidden reference files, and solver-verified optima from one structured instance record, scored by executing generated solver code against the verified optimum, with a two-stage diagnostic for reading vs downstream failures.
Key results
- GPT-5.4 reaches 52.1% pass@1 and Gemini 3.1 Pro Preview 51.3% on MM-OptBench.
- Average pass@1 across six general-purpose MLLMs falls from 43.4% on easy to 15.9% on hard instances.
- All three math-specialized MLLMs solve 0/780 instances under official scoring.
Surprising findings
- Executable code is common (about 92% valid code rate for top models) yet roughly half of solutions fail solver-grounded verification.
- Math-specialized MLLMs obtain zero official solves and still solve almost no cases even with oracle instance data.
Limitations
- Dataset and benchmark code are not released at this stage; release promised upon acceptance.
- Stage-2 extraction diagnostics only partly resolve failure causes (5.2% unresolved).
Experimental setup
Benchmarks: MM-OptBench
Metrics: pass@1 pass@4 Valid Code Rate
Baselines: MathCoder-VL-8B MM-Eureka MM-PRM
Builds on: NL4Opt IndustryOR ComplexOR OptMath MIPLIB-NL
Closest papers by meaning
- PEARL: Solver-in-the-Loop Interactive Optimization Modeling from Natural Language (NeurIPS 2026 · similarity 0.90)
- OptiBench Meets ReSocratic: Measure and Improve LLMs for Optimization Modeling (ICLR 2025 · similarity 0.90)
- OptMATH: A Scalable Bidirectional Data Synthesis Framework for Optimization Modeling (ICML 2025 · similarity 0.89)
- LLMOPT: Learning to Define and Solve General Optimization Problems from Scratch (ICLR 2025 · similarity 0.89)
- SolverLLM: Leveraging Test-Time Scaling for Optimization Problem via LLM-Guided Search (NeurIPS 2025 · similarity 0.88)
- Autoformulation of Mathematical Optimization Models Using LLMs (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.