Research map / Large language models / Agents, code and math
OptiBench Meets ReSocratic: Measure and Improve LLMs for Optimization Modeling
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces OptiBench, an end-to-end optimization-modeling benchmark (605 problems, linear/nonlinear, with/without tables) and ReSocratic, a reverse-Socratic data synthesis method that yields the 29k-example ReSocratic-29k dataset for SFT of open-source models.
Problem
Existing optimization-modeling benchmarks focus on linear programs, omit tabular inputs and end-to-end numerical solving, and open-source small LLMs lack optimization training data.
Approach
Synthesize formatted optimization demonstrations step by step (variables, objective, constraints with math formalization) from seed examples, then back-translate them into natural-language questions and generate solver code, instead of forward question-then-answer synthesis.
Key results
- Llama-3-8B-Instruct improves from 13.6% (few-shot) to 51.1% overall accuracy on OptiBench after SFT on ReSocratic-29k.
- Llama-2-7B-Chat improves from 0.0% to 30.6% after SFT on ReSocratic-29k.
- GPT-4o reaches 69.4% overall few-shot accuracy, the highest reported on OptiBench.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| SFT on ReSocratic-29k | Llama-2-7B-Chat · OptiBench | accuracy: 0.0% to 30.6% | Helped |
| SFT on ReSocratic-29k | Llama-3-8B-Instruct · OptiBench | accuracy: 13.6% to 51.1% | Helped |
| add step questions to ReSocratic training data (ablation) | Llama-2-7B-Chat · OptiBench SFT | accuracy: 28.9% -> 30.6% | Helped |
| add step questions to ReSocratic training data (ablation) | Llama-3-8B-Instruct · OptiBench SFT | accuracy: 50.2% -> 51.1% | Helped |
| apply rule and similarity filters to ReSocratic data (ablation) | Llama-2-7B-Chat · OptiBench SFT | accuracy: 29.6% -> 30.6% | Helped |
| apply rule and similarity filters to ReSocratic data (ablation) | Llama-3-8B-Instruct · OptiBench SFT | accuracy: 49.4% -> 51.1% | Helped |
| use reverse ReSocratic synthesis instead of forward Self-Instruct synthesis | Llama-2-7B-Chat · OptiBench SFT, 1k responses | accuracy: 10.7% -> 14.4% | Helped |
| use reverse ReSocratic synthesis instead of forward Evol-Instruct synthesis | Llama-2-7B-Chat · OptiBench SFT, 1k responses | accuracy: 11.1% -> 14.4% | Helped |
| generate reasoning before writing code in the few-shot prompt | GPT-3.5-Turbo · OptiBench few-shot | overall accuracy: 56.4 -> 55.5 (-0.9) | Hurt |
| generate reasoning before writing code in the few-shot prompt | GPT-4 · OptiBench few-shot | overall accuracy: 65.5 -> 63.8 (-1.7) | Hurt |
| generate reasoning before writing code in the few-shot prompt | GPT-4o-mini · OptiBench few-shot | overall accuracy: 55.0 -> 54.9 (-0.1) | No clear effect |
| generate reasoning before writing code in the few-shot prompt | GPT-4o · OptiBench few-shot | overall accuracy: 69.4 -> 67.1 (-2.3) | Hurt |
Surprising findings
- Few-shot first-reasoning-then-code prompting does not improve over the original few-shot prompt for GPT models.
- Llama-3-8B-Instruct after SFT exceeds GPT-4 on the linear with-table subset.
Limitations
- Small seed pool of 27 demonstrations and reliance on DeepSeek-V2 for synthesis.
- Benchmark limited to explicit-number problems solved with PySCIPOpt; nonlinear and tabular subsets are small and results on them are noisy.
- Improvements over closed-source models remain limited for nonlinear and table-based problems.
Experimental setup
Datasets: ReSocratic-29k NL4OPT ComplexOR NLP4LP MAMO
Benchmarks: OptiBench
Metrics: answer accuracy code pass rate pass@k
Baselines: GPT-4 GPT-4o GPT-3.5-Turbo Llama-3-70B-Instruct DeepSeek-V2 DeepSeek-V2.5 Qwen2-7B-Instruct Mistral-7B-Instruct-v0.3 Self-Instruct Evol-Instruct WizardLM
Builds on: Rejection Sampling Fine-Tuning GSM8K Socratic dataset WizardLM MAmmoTH MetaMath
Closest papers by meaning
- LLMOPT: Learning to Define and Solve General Optimization Problems from Scratch (ICLR 2025 · similarity 0.90)
- MM-OptBench: A Solver-Grounded Benchmark for Multimodal Optimization Modeling (NeurIPS 2026 · similarity 0.90)
- OptiMUS: Scalable Optimization Modeling with (MI)LP Solvers and Large Language Models (ICML 2024 · similarity 0.89)
- OptMATH: A Scalable Bidirectional Data Synthesis Framework for Optimization Modeling (ICML 2025 · similarity 0.89)
- Autoformulation of Mathematical Optimization Models Using LLMs (ICML 2025 · similarity 0.88)
- Step-by-Step Optimization-like Reasoning in LLMs over Expanding Search Spaces (NeurIPS 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.