Research map / Large language models / Agents, code and math

DiagnosticIQ: A Benchmark for LLM-Based Industrial Maintenance Action Recommendation from Symbolic Rules

NeurIPS 2026 · Large language models › Agents, code and math · approach: Code generation benchmarks · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces DiagnosticIQ, a 6,690-question MCQA benchmark that turns expert-authored industrial alarm rules into maintenance-action questions, and evaluates 29 LLMs plus embedding baselines with robustness and logical-inversion variants.

Problem

LLMs' ability to recommend the correct corrective maintenance action once an engineer-authored symbolic sensor rule fires has no standardized benchmark, and existing industrial QA benchmarks test fault recognition rather than action selection.

Approach

A deterministic symbolic-to-MCQA pipeline that normalizes rule condition trees to DNF, expands each conjunction into a trigger scenario, and samples distractors by rule-rule embedding similarity (selection vs. elimination question types), plus variants (Pro, Pert, Verbose, Aug, Rationale) that probe specific failure modes.

Key results

• Claude-opus-4-6 leads with 73.59% Macro on DiagnosticIQ, with gpt-5.4 (73.27%) and gemini-3.1-pro-preview (72.74%) within one point; on Pro, accuracy drops to 59.81%.
• Under condition inversion (Aug), claude-opus-4-6 selects the original (now incorrect) answer 63.43% of the time versus 20.73% for 'None of the above'.
• Paraphrasing symbolic conditions (Verbose) reduces Macro accuracy by 4.2 points on average across 17 models, up to 9.6 points.

Surprising findings

• Embedding-only cosine similarity baseline (all-mpnet-base-v2) surpasses 14 of 29 LLMs on DiagnosticIQ Pro.
• Frontier models select the originally correct answer under logical inversion more often than the correct 'None of the above' option, showing template-matching behavior.

Limitations

• Scope limited to English-language rules under zero-shot evaluation.
• Human evaluation and rationale analysis rely on a small expert panel (9 practitioners; 5 experts for rationales) with moderate inter-rater agreement.
• Dataset heavily skewed toward AHU assets (58%) and derived from a single rule corpus (Smarter Buildings).

Experimental setup

Datasets: DiagnosticIQ rules from Smarter Buildings initiative 28 rules from an AHU-focused work (extension)

Benchmarks: DiagnosticIQ DiagnosticIQ Pro DiagnosticIQ Pert DiagnosticIQ Verbose DiagnosticIQ Aug DiagnosticIQ Rationale

Metrics: Macro accuracy Micro accuracy consistency relative accuracy drop Bradley-Terry Elo estimated cost savings

Baselines: all-mpnet-base-v2 multi-qa-mpnet minilm-l6 distilroberta claude-opus-4-6 gpt-5.4 gemini-3.1-pro-preview claude-3-7-sonnet

Builds on: MMLU-Pro PertEval FailureSensorIQ GRPO

Closest papers by meaning

• FailureSensorIQ: A Multi-Choice QA Dataset for Understanding Sensor Relationships and Failure Modes (NeurIPS 2025 · similarity 0.84)
• MMAD: A Comprehensive Benchmark for Multimodal Large Language Models in Industrial Anomaly Detection (ICLR 2025 · similarity 0.83)
• Judo: A Juxtaposed Domain-oriented Multimodal Reasoner for Industrial Anomaly QA (ICLR 2026 · similarity 0.82)
• AD-FM: Multimodal LLMs for Anomaly Detection via Multi-Stage Reasoning and Fine-Grained Reward Optimization (AAAI 2026 · similarity 0.82)
• InCLAD: A Continual Learning Benchmark for Industrial Visual Anomaly Detection (NeurIPS 2026 · similarity 0.82)
• Where Root Cause Analysis Fails: A Retrieval-Reranking Decomposition (NeurIPS 2026 · similarity 0.82)