Research map / Large language models / Agents, code and math
OptMATH: A Scalable Bidirectional Data Synthesis Framework for Optimization Modeling
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes OptMATH, a bidirectional data synthesis framework that generates optimization problem data with controllable complexity, back-translates it to natural language, and validates via forward modeling and optimal-value rejection sampling, then fine-tunes Qwen2.5 models (0.5B-32B) as AutoFormulator.
Problem
Optimization modeling (translating natural-language optimization problems into solver-ready formulations) is limited by the scarcity of high-quality, verified (NL, MF, PD) training data and by weak generalization of learning-based methods.
Approach
Generate solver-level problem instances from 53 seed generators with LLM-tuned complexity controls, back-translate them to NL with an iterative self-critique/refine pipeline, and keep only NL-to-model pairs whose solved optimal values match the originals (rejection sampling), using the accepted pairs to fine-tune an optimization formulator and the rejected hard pairs as a benchmark.
Key results
- OptMATH-Qwen2.5-32B reaches 95.9% on NL4OPT, 89.9% on MAMO EasyLP, 54.1% on MAMO ComplexLP and 34.7% on OptMATH-Bench, outperforming GPT-4 and DeepSeek-V3 on the reported benchmarks.
- OptMATH-Qwen2.5-7B reaches 64.2% macro average versus 51.4% for ORLM-LLaMA-3-8B, which gets 0.0% on OptMATH-Bench.
- Manual analysis of 1% of the dataset found 99.6% equivalence accuracy for rejection-sampled triplets; the self-refine loop with T=1 gave a 61.56% acceptance rate.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune Qwen2.5-32B on OptMATH-Train (ablation) | Qwen2.5-32B · NL4OPT | pass@1 accuracy: 92.65% -> 95.9% | Helped |
| Fine-tune Qwen2.5-32B on OptMATH-Train (ablation) | Qwen2.5-32B · MAMO EasyLP | pass@1 accuracy: 82.21% -> 90.49% | Helped |
| Fine-tune Qwen2.5-32B on OptMATH-Train (ablation) | Qwen2.5-32B · MAMO ComplexLP | pass@1 accuracy: 44.55% -> 52.6% | Helped |
| Fine-tune Qwen2.5-32B on OptMATH-Train (ablation) | Qwen2.5-32B · OptMATH-Bench | pass@1 accuracy: 9.33% -> 30.6% | Helped |
| Use mixed original and augmented training data instead of raw data only (ablation) | Qwen2.5-7B · NL4OPT | pass@1 accuracy: 86.9% -> 87.3% | Helped |
| Use mixed original and augmented training data instead of raw data only (ablation) | Qwen2.5-7B · MAMO EasyLP | pass@1 accuracy: 88.0% -> 87.7% | Hurt |
| Use mixed original and augmented training data instead of raw data only (ablation) | Qwen2.5-7B · MAMO ComplexLP | pass@1 accuracy: 44.5% -> 48.1% | Helped |
| Use mixed original and augmented training data instead of raw data only (ablation) | Qwen2.5-7B · OptMATH-Bench | pass@1 accuracy: 31.1% -> 33.3% | Helped |
| Train with augmented data only rather than raw data (ablation) | Qwen2.5-7B · MAMO ComplexLP | pass@1 accuracy: 44.5% -> 44.7% | Helped |
| Train with augmented data only rather than raw data (ablation) | Qwen2.5-7B · NL4OPT | pass@1 accuracy: 86.9% -> 82.9% | Hurt |
| Add the Self-Refine loop to direct generation (ablation) | OptMATH backtranslation pipeline · OptMATH-Train synthesis sample | acceptance rate: T=1 achieves a satisfactory acceptance rate of 61.56% | Helped |
| Increase the OptMATH-Train data used for fine-tuning (ablation) | Qwen2.5 models · Optimization modeling benchmarks | pass@1 accuracy | Mixed |
Surprising findings
- Fine-tuning Qwen2.5-7B on augmented data alone gave better accuracy on hard problems but worse on simple ones; a 50/50 mix gave the best average.
- Relative gains from OptMATH-Train fine-tuning show diminishing returns as model size grows, and the 32B base model scores lower than the 14B base on OptMATH-Bench in the size ablation.
Limitations
- Solution-based rejection sampling (matching optimal values) may accept non-equivalent formulations; equivalence was checked only manually on 1% of the data.
- Performance on the hard OptMATH-Bench remains modest (about 35% for the best model), and larger models gain little on complex benchmarks.
- Backtranslation and generation rely on DeepSeek-V3 and an LLM-based feedback loop, which adds dependence on a proprietary-scale teacher model.
Experimental setup
Datasets: MIPLIB 2017 netlib NL4OPT MAMO EasyLP MAMO ComplexLP
Benchmarks: NL4OPT MAMO EasyLP MAMO ComplexLP OptMATH-Bench
Metrics: pass@1 accuracy micro average macro average acceptance rate
Baselines: GPT-3.5-turbo GPT-4 Deepseek-V3 Chain-of-Experts Optimus ORLM-LLaMA-3-8B
Builds on: OR-Instruct self-refine chain-of-thought LoRA rejection sampling
Closest papers by meaning
- LLMOPT: Learning to Define and Solve General Optimization Problems from Scratch (ICLR 2025 · similarity 0.89)
- Autoformulation of Mathematical Optimization Models Using LLMs (ICML 2025 · similarity 0.89)
- MM-OptBench: A Solver-Grounded Benchmark for Multimodal Optimization Modeling (NeurIPS 2026 · similarity 0.89)
- OptiBench Meets ReSocratic: Measure and Improve LLMs for Optimization Modeling (ICLR 2025 · similarity 0.89)
- EvoOptiGraph: Weakness-Driven Coevolution via Graph-Based Structural Generation for Optimization Modeling (NeurIPS 2026 · similarity 0.88)
- SolverLLM: Leveraging Test-Time Scaling for Optimization Problem via LLM-Guided Search (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.