Research map / Large language models / Agents, code and math

Solver-Informed RL: Grounding Large Language Models for Authentic Optimization Modeling

NeurIPS 2025 · Large language models › Agents, code and math · approach: LLMs for optimization and heuristics · 2 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes SIRL, an RLVR framework where solver feedback (execution, feasibility, objective accuracy, .lp-based bonus) rewards LLMs for optimization modeling, plus an instance-enhanced self-consistency method for data synthesis and a Partial KL surrogate.

Problem

LLMs generating optimization models from natural language often produce non-executable, infeasible, or incorrect formulations, and offline SFT/preference methods do not guarantee functional correctness.

Approach

Use an optimization solver and the generated .lp file as verifiable reward sources for online RL (REINFORCE++ with a Partial KL penalty applied only to formulation and code segments), and use solver-derived instance features (objective value, direction, binary/integer counts) in a weighted self-consistency vote to build training data.

Key results

• SIRL-Qwen2.5-32B achieves 68.3% macro average accuracy across five benchmarks, above DeepSeek-V3.1 (65.0%), DeepSeek-R1 (64.6%) and OpenAI-o3 (57.1%).
• SIRL-Qwen2.5-7B reaches 60.6% macro average, above OptMATH-Qwen2.5-7B (55.4%) and ORLM-LLaMA-3-8B (46.4%).
• Instance-enhanced self-consistency improves over value-based voting, e.g. OptMATH 32B pass@10 rises from 27.5% to 34.2%.

Measured effects reported in the paper

Surprising findings

• Pure textual reasoning without solver execution yields very low accuracy (e.g., 1.4% on MAMO Complex for Qwen2.5-7B-Instruct), indicating solver use is essential.
• Removing KL regularization entirely lowers execution rate substantially (e.g., 87.0% vs 96.0% on IndustryOR), though it helps reasoning diversity.

Limitations

• Reward hacking persists according to the authors.
• Performance remains limited on harder benchmarks such as IndustryOR and OptMATH.
• Benchmark test sets were manually corrected by the authors, which may make comparisons with originally reported baselines inconsistent.

Experimental setup

Datasets: NL4OPT MAMO IndustryOR OptMATH synthetic OR-Instruct data from seed cases

Benchmarks: NL4OPT MAMO Easy MAMO Complex IndustryOR OptMATH

Metrics: pass@1 accuracy macro average execution rate

Baselines: GPT-4 DeepSeek-V3.1 DeepSeek-R1 OpenAI-o3 OptiMUS ORLM-LLaMA-3-8B LLMOpt-Qwen2.5-14B OptMATH-Qwen2.5-7B OptMATH-Qwen2.5-32B

Builds on: REINFORCE++ OR-Instruct (ORLM) GRPO-style RLVR self-consistency Verl

Closest papers by meaning

• $\mathcal{B}$-Coder: Value-Based Deep Reinforcement Learning for Program Synthesis (ICLR 2024 · similarity 0.89)
• Prompted Policy Search: Reinforcement Learning through Linguistic and Numerical Reasoning in LLMs (NeurIPS 2025 · similarity 0.88)
• Large Language Models as End-to-end Combinatorial Optimization Solvers (NeurIPS 2025 · similarity 0.88)
• SolverLLM: Leveraging Test-Time Scaling for Optimization Problem via LLM-Guided Search (NeurIPS 2025 · similarity 0.88)
• PEARL: Solver-in-the-Loop Interactive Optimization Modeling from Natural Language (NeurIPS 2026 · similarity 0.88)
• MM-OptBench: A Solver-Grounded Benchmark for Multimodal Optimization Modeling (NeurIPS 2026 · similarity 0.88)