Research map / Large language models / Agents, code and math
AutoOpt: A Dataset and a Unified Framework for Automating Optimization Problem Solving
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces AutoOpt-11k, an 11.5k-image dataset of handwritten and printed optimization formulations with LaTeX (and PYOMO for a 1,018 subset) labels, and the AutoOpt pipeline (image-to-LaTeX hybrid ResNet-Swin/mBART model, fine-tuned DeepSeek-Coder LaTeX-to-PYOMO, and a bilevel decomposition solver).
Problem
Automating the solution of optimization problems given only as images (handwritten or printed), which requires recognizing the formulation, converting it to a modeling language, and solving it; no image-to-optimization-model dataset existed.
Approach
A new expert-annotated image-to-LaTeX-to-PYOMO optimization dataset plus a three-module pipeline: a hybrid ResNet+Swin encoder with mBART decoder (initialized from Nougat) for image-to-LaTeX, a fine-tuned small code LLM for LaTeX-to-PYOMO, and a BOBD solver that uses a GA at the upper level, an LR-based variable classifier, and a convex/classical solver at the lower level.
Key results
- AutoOpt-M1 reports handwritten CER 0.0412 and handwritten+printed CER 0.0286, beating GPT-4o (0.1465 / 0.1017), Gemini 2.0 Flash, and Nougat on BLEU and CER.
- Fine-tuned DeepSeek-Coder 1.3B (AutoOpt-M2) reaches BLEU 88.25 and CER 0.0825 on LaTeX-to-PYOMO.
- Framework-level success rate on 500 out-of-dataset problems is 94.20%, with estimated module-chain reliability of 89.12%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Combine CNN and Transformer in the vision encoder, rather than using Transformer alone (ablation) | AutoOpt-M1 · AutoOpt-11k test | BLEU: 95.51 -> 96.70 | Helped |
| Combine CNN and Transformer in the vision encoder, rather than using Transformer alone (ablation) | AutoOpt-M1 · AutoOpt-11k test | CER: 0.0440 -> 0.0286 | Helped |
| Combine CNN and Transformer in the vision encoder, rather than using CNN alone (ablation) | AutoOpt-M1 · AutoOpt-11k test | BLEU: 16.10 -> 96.70 | Helped |
| Combine CNN and Transformer in the vision encoder, rather than using CNN alone (ablation) | AutoOpt-M1 · AutoOpt-11k test | CER: 0.8812 -> 0.0286 | Helped |
| Use AutoOpt-M1 instead of fine-tuned Nougat | MER · AutoOpt-11k HW test | CER: 0.0752 -> 0.0412 | Helped |
| Use AutoOpt-M1 instead of fine-tuned Nougat | MER · AutoOpt-11k PR test | CER: 0.0168 -> 0.0176 | Hurt |
| Use AutoOpt-M1 instead of fine-tuned Nougat | MER · AutoOpt-11k HW+PR test | CER: 0.0440 -> 0.0286 | Helped |
| Use AutoOpt-M1 instead of GPT 4o | MER · AutoOpt-11k HW+PR test | CER: 0.1017 -> 0.0286 | Helped |
| Use AutoOpt-M1 instead of Gemini 2.0 Flash | MER · AutoOpt-11k HW+PR test | CER: 0.1338 -> 0.0286 | Helped |
Surprising findings
- Adding the CNN branch to a transformer-only encoder raised BLEU only modestly (95.51 to 96.70), while CNN-only performed very poorly (BLEU 16.10).
- Nougat slightly beats AutoOpt-M1 on printed-only CER (0.0168 vs 0.0176).
Limitations
- Framework does not handle ill-defined optimization problems.
- Does not handle problem definitions spanning multiple pages or images.
- BOBD is not claimed to beat specialized optimization solvers on optimality or convergence rates.
Experimental setup
Datasets: AutoOpt-11k
Benchmarks: AutoOpt-11k test split 10 optimization test problems TP1-TP10
Metrics: BLEU character error rate success rate
Baselines: GPT-4o Gemini 2.0 Flash Nougat interior-point algorithm genetic algorithm DL1 (CNN only) DL2 (Transformer only)
Builds on: Nougat Donut Swin Transformer mBART DeepSeek-Coder
Closest papers by meaning
- MM-OptBench: A Solver-Grounded Benchmark for Multimodal Optimization Modeling (NeurIPS 2026 · similarity 0.86)
- Autoformulation of Mathematical Optimization Models Using LLMs (ICML 2025 · similarity 0.85)
- B2Opt: Learning to Optimize Black-box Optimization with Little Budget (AAAI 2025 · similarity 0.85)
- SYMBOL: Generating Flexible Black-Box Optimizers through Symbolic Equation Learning (ICLR 2024 · similarity 0.85)
- BBOWP-Bench: Evaluating LLMs on Black-Box Optimization Word Problems (NeurIPS 2026 · similarity 0.84)
- MetaBox-v2: A Unified Benchmark Platform for Meta-Black-Box Optimization (NeurIPS 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.