Research map / Large language models / Reasoning and chain of thought

StepORLM: A Self-Evolving Framework With Generative Process Supervision For Operations Research Language Models

ICLR 2026 · Large language models › Reasoning and chain of thought · approach: RL with verifiable rewards for reasoning

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes StepORLM, a self-evolving framework where an 8B policy (Qwen3-8B) and a generative process reward model co-evolve using solver outcome checks and GenPRM step critiques, aligned via weighted DPO, reaching SOTA on six OR benchmarks.

Problem

Outcome-only RL for LLM operations research modeling suffers credit assignment problems, while discriminative process reward models are myopic and evaluate steps in isolation rather than holistically.

Approach

A co-evolutionary loop combining solver-verified outcome labels with generative (chain-of-thought, trajectory-level) process critiques from a GenPRM; preference pairs are weighted (W-DPO) and the GenPRM is refined by SFT on step labels from solver-verified trajectories.

Key results

• StepORLM (8B) reaches 79.0% macro-average Pass@1 versus 65.0% for ORLM (8B) and 75.4% for DeepSeek-V3 (671B) in the main table.
• StepORLM+GenPRM reaches 85.6% average, with gains of 9.9 points on ComplexOR and 9.5 points on IndustryOR over StepORLM alone.
• Using GenPRM as verifier raises ORLM's average from 65.0% to 75.0%, a roughly 10-point gain on a different policy model.

Surprising findings

• The paper reports a non-monotonic trend on IndustryOR, where accuracy first declines before increasing at the third iteration.
• Discriminative PRM inference scaling gives only small gains (e.g., 82.2% vs 81.4% for StepORLM), while GenPRM gives larger gains.

Limitations

• Small test subsets (e.g., 18 ComplexOR and 42 IndustryOR problems) make results noisy, as the authors note for IndustryOR.
• Training data is synthesized by GPT-4o and the process supervision relies on the generative GenPRM's critiques, which may be imperfect.
• Inference scaling with GenPRM requires generating multiple trajectories and extra critique calls, adding cost.

Experimental setup

Datasets: NL4Opt MAMO EasyLP MAMO ComplexLP NLP4LP ComplexOR IndustryOR ReSocratic

Benchmarks: NL4Opt MAMO EasyLP MAMO ComplexLP NLP4LP ComplexOR IndustryOR ReSocratic

Metrics: Pass@1 accuracy macro-average accuracy

Baselines: GPT-4o DeepSeek-V3 Qwen3-32B Qwen2.5-72B-Instruct ORLM LLMOPT OptMATH OptiMUS-v0.3 CoT CoE CAFA

Builds on: Direct Preference Optimization Solver-Informed RL process reward models

Closest papers by meaning

• GenPRM: Scaling Test-Time Compute of Process Reward Models via Generative Reasoning (AAAI 2026 · similarity 0.91)
• Self-evolving LLM agents with in-distribution Optimization (ICML 2026 · similarity 0.90)
• Rewarding Progress: Scaling Automated Process Verifiers for LLM Reasoning (ICLR 2025 · similarity 0.89)
• Do We Need to Verify Step by Step? Rethinking Process Supervision from a Theoretical Perspective (ICML 2025 · similarity 0.88)
• STRIDE: Learnable Stepwise Language Feedback for LLM Reasoning (NeurIPS 2026 · similarity 0.87)
• ReST-MCTS*: LLM Self-Training via Process Reward Guided Tree Search (NeurIPS 2024 · similarity 0.87)