Research map / Large language models / Agents, code and math
Evaluating LLM Reasoning in the Operations Research Domain with ORQA
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces ORQA, a 1,513-instance expert-crafted multiple-choice benchmark for operations research reasoning, and evaluates open-source LLMs under standard and CoT prompting, finding modest performance and that CoT often hurts.
Problem
LLMs' ability to generalize expert-level knowledge and multi-step reasoning to the specialized technical domain of Operations Research (identifying optimization model components and relations) is unclear, and existing OR datasets are small, toy-like, or evaluated only via solver output.
Approach
A solver-free, multiple-choice QA benchmark of expert-written optimization problem descriptions with questions on model components, relations, and logic, with options crafted by OR experts to be challenging, evaluated under standard and two-step CoT prompting in zero- and few-shot settings.
Key results
- Llama3.1-405B-Instruct achieved the highest accuracy of 0.772 on ORQA with 3-shot standard prompting.
- CoT prompting generally reduced accuracy; e.g., Llama3.1-70B-Instruct dropped from 0.735 (3-shot standard) to 0.292 (1-shot CoT), with 0-shot CoT at 0.689 vs 0.702 standard.
- A single OR PhD expert achieved 93% accuracy on a 100-instance subset without ICL examples.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase model size within the Llama 3.1 family | Llama3.1-Instruct · ORQA test, Standard 3-shot | accuracy: 0.618 -> 0.735 -> 0.772 | Helped |
| Use CoT instead of standard prompting | LLMs on ORQA · ORQA test | accuracy: Llama3.1-405B-I, 3-shot Standard: 0.772; CoT 0-shot: 0.695; CoT 1-shot: 0.360 | Hurt |
| Add standard in-context examples | LLMs on ORQA · ORQA test | accuracy: Llama3.1-405B-I: 0.723 -> 0.753 -> 0.772 | Helped |
| Add one in-context example to CoT prompting | Llama3.1-405B-I · ORQA test | accuracy: 0.695 -> 0.360 | Hurt |
| Use an OR-expert trigger prompt rather than “Let's work by elimination” (ablation) | Llama3.1-70B-I, 0-shot CoT · ORQA test | accuracy: 0.648 -> 0.689 | Helped |
| Ensemble all trigger prompts using majority vote (ablation) | Llama3.1-70B-I, 0-shot CoT · ORQA test | accuracy: 0.689 -> 0.696 | Helped |
| Select an ICL example with the same question type instead of randomly (ablation) | Llama2-13B-Chat, 1-shot CoT · ORQA | accuracy: 0.300 -> 0.313 | Helped |
| Select an ICL example with similar prompt length instead of randomly (ablation) | Llama2-13B-Chat, 1-shot CoT · ORQA | accuracy: 0.300 -> 0.353 | Helped |
| Select an ICL example with similar length and the same question type instead of randomly (ablation) | Llama2-13B-Chat, 1-shot CoT · ORQA | accuracy: 0.300 -> 0.362 | Helped |
| Add same-question-type selection to similar-length ICL selection (ablation) | Llama2-13B-Chat, 1-shot CoT · ORQA | accuracy: 0.353 -> 0.362 | Helped |
Surprising findings
- Chain-of-thought prompting generally decreased performance on ORQA, and ICL examples helped standard prompting but not CoT.
- Models often ignored the single-answer instruction in CoT reasoning and hallucinated options.
Limitations
- Evaluation restricted to open-source models; closed-source models like GPT-4 not evaluated by choice.
- Small validation set (45 instances) used for few-shot examples and only 100 instances for human baseline.
- Focuses on multiple-choice component identification rather than full model formulation or solving.
Experimental setup
Datasets: ORQA
Benchmarks: ORQA
Metrics: accuracy macro F1
Baselines: Llama3.1-405B-Instruct Llama3.1-70B-Instruct Llama3-70B-Instruct Mixtral-8x7B-Instruct-v0.1 Mistral-7B-Instruct-v0.3 FLAN-T5-XXL Falcon-7B-Instruct NuminaMath-7B DeepSeek-Math-7B-Instruct
Builds on: chain-of-thought prompting NLP4LP ComplexOR
Closest papers by meaning
- SuperGPQA: Scaling LLM Evaluation across 285 Graduate Disciplines (NeurIPS 2025 · similarity 0.89)
- GPO: Learning from Critical Steps to Improve LLM Reasoning (NeurIPS 2025 · similarity 0.89)
- GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models (ICLR 2025 · similarity 0.89)
- GLoRe: When, Where, and How to Improve LLM Reasoning via Global and Local Refinements (ICML 2024 · similarity 0.88)
- Explore What LLM Does Not Know in Complex Question Answering (AAAI 2025 · similarity 0.88)
- QuestBench: Can LLMs ask the right question to acquire information in reasoning tasks? (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.