Research map / Large language models / AI and society
Decision Information Meets Large Language Models: The Future of Explainable Operations Research
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Explainable Operations Research (EOR), a multi-agent LLM framework that answers what-if queries by modifying OR models and quantifying changes via bipartite-graph edit distance, plus a new 30-problem industrial explanation benchmark built on IndustryOR.
Problem
Existing LLM-based OR methods focus on modeling accuracy and offer only shallow explanations, and prior explainable-OR work (OptiGuide) handles only easy what-if changes such as parameter edits, not complex constraint additions/deletions with quantitative justification.
Approach
Define "Decision Information" as parameters and constraints in a user query; convert original and modified LPs into bipartite graphs and compute normalized graph edit distance to quantify the impact of changes, with a Commander/Writer/Safeguard agent workflow generating attribution and justification explanations.
Key results
- Zero-shot GPT-4-Turbo modeling accuracy: EOR 88.33% vs Standard 63.00% and OptiGuide 30.33%.
- One-shot GPT-4-Turbo modeling accuracy reaches 95.33% with EOR, versus 88.00% Standard and 69.33% OptiGuide.
- Expert-scored overall explanation quality for EOR (zero-shot) is 9.47/10 vs 5.63 for Standard and 6.59 for OptiGuide.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use EOR instead of the strongest baseline | GPT-4-Turbo · IndustryOR zero-shot | accuracy: 63.00% -> 88.33% | Helped |
| Use EOR instead of the strongest baseline | GPT-4-Turbo · IndustryOR one-shot | accuracy: 88.00% -> 95.33% | Helped |
| Provide one example rather than zero-shot prompting | EOR with GPT-4-Turbo · IndustryOR | accuracy: 88.33% -> 95.33% | Helped |
| Use EOR instead of Standard or OptiGuide | GPT-4-Turbo · IndustryOR zero-shot | Overall Expert: Standard 5.63; OptiGuide 6.59; EOR 9.47 | Helped |
| Increase temperature from 0 to 0.5 (ablation) | EOR with GPT-4-Turbo · IndustryOR zero-shot | accuracy: 88.33% -> 89.67% | No clear effect |
| Increase temperature from 0 to 1 (ablation) | EOR with GPT-4-Turbo · IndustryOR zero-shot | accuracy: 88.33% -> 88.00% | No clear effect |
| Increase temperature from 0 to 0.5 (ablation) | EOR with GPT-4-Turbo · IndustryOR one-shot | accuracy: 95.33% -> 93.67% | Hurt |
| Increase temperature from 0 to 1 (ablation) | EOR with GPT-4-Turbo · IndustryOR one-shot | accuracy: 95.33% -> 93.33% | Hurt |
| Increase debug attempts from 3 to 10 (ablation) | EOR with GPT-4-Turbo · IndustryOR zero-shot | accuracy: 88.33% -> 89.00% | No clear effect |
| Increase debug attempts from 3 to 10 (ablation) | EOR with GPT-4-Turbo · IndustryOR one-shot | accuracy: 95.33% -> 95.33% | No clear effect |
Surprising findings
- Standard outperforms OptiGuide for all LLMs except GPT-4 in zero-shot, and OptiGuide's accuracy jumps from 30.33% to 75.00% on GPT-4 with one example.
- Increasing debug attempts from 3 to 10 does not meaningfully improve accuracy, suggesting limited self-correction.
Limitations
- Benchmark is small (30 problems, 300 queries) and built from a single source (IndustryOR).
- Automated explanation scoring relies on a single LLM judge (GPT-4o) that may be biased toward LLM-generated outputs.
- Failure modes include modeling logic errors and JSON format errors, and debugging loops show limited self-correction.
Experimental setup
Datasets: IndustryOR
Benchmarks: EOR explanation benchmark
Metrics: modeling accuracy explanation quality score (EC, ER, Overall, 0-10)
Baselines: Standard (direct GPT prompting) OptiGuide
Builds on: OptiGuide IndustryOR AutoGen
Closest papers by meaning
- Enhancing Decision Making Through the Integration of Large Language Models and Operations Research Optimization (AAAI 2025 · similarity 0.87)
- A Framework for Data-Driven Explainability in Mathematical Optimization (AAAI 2024 · similarity 0.85)
- DeLLMa: Decision Making Under Uncertainty with Large Language Models (ICLR 2025 · similarity 0.84)
- Evaluating LLM Reasoning in the Operations Research Domain with ORQA (AAAI 2025 · similarity 0.84)
- Navigating the Maze of Explainable AI: A Systematic Approach to Evaluating Methods and Metrics (NeurIPS 2024 · similarity 0.84)
- Rethinking XAI Evaluation: A Human-Centered Audit of Shapley Benchmarks in High-Stakes Settings (NeurIPS 2026 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.