Research map / Large language models / Retrieval and knowledge
FailureSensorIQ: A Multi-Choice QA Dataset for Understanding Sensor Relationships and Failure Modes
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces FailureSensorIQ, a template-generated MCQA benchmark (8,296 questions over 10 industrial assets from ISO documents) testing LLM knowledge of failure mode–sensor relationships, evaluated with perturbation, uncertainty, option-complexity, agent, and expert studies.
Problem
LLMs' specialized knowledge of industrial assets, failure modes and sensor relationships (for condition-based maintenance and FMEA) is untested; existing QA benchmarks do not cover this domain.
Approach
Build a non-LLM, template-based MCQA generator from non-textual ISO failure-mode/sensor mapping tables, in single-correct and multi-correct, selection and elimination formats, then stress-test LLMs with perturbations, distractor expansion, uncertainty set sizes, and a ReAct agent with Wikipedia/arXiv.
Key results
- o1 achieves 60.40% accuracy on single-correct questions (SC-MCQA), but drops to 44.17% consistency-based accuracy under perturbation; ACC@Perturb falls to 49.76%.
- With 10 options (OptionsPert), o1's accuracy falls from 60% to 5.74%.
- Expert human participants reached a maximum of 66.19% and mean 60.20% accuracy; a practitioner lagged experts by over 25 points on average.
- ReAct agent with Wikipedia and arXiv lowered llama-3.3-70b accuracy from 41.69% to 37.57%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use standard chain-of-thought prompting instead of direct prompting (ablation) | Llama 3.3 70B · FailureSensorIQ SC-MCQA | accuracy: 41.69 to 51.18 | Helped |
| Use Plan@Solve prompting instead of direct prompting (ablation) | Llama 3.1 8B · FailureSensorIQ SC-MCQA | accuracy: 40.04 to 46.46 | Helped |
| Use chain-of-thought prompting (ablation) | Llama models · FailureSensorIQ SC-MCQA | accuracy | Mixed |
| Add Wikipedia and arXiv retrieval with ReAct (ablation) | Llama 4 Maverick 17B 128E · FailureSensorIQ SC-MCQA | accuracy: 55.83% to 48.44% | Hurt |
| Add Wikipedia and arXiv retrieval with ReAct (ablation) | Llama models · FailureSensorIQ SC-MCQA | correct response rate: 41.69% to 37.57%; 40.04% to 37.05% | Mixed |
| Add distractor options in OptionsPert (ablation) | o1 · FailureSensorIQ OptionsPert | accuracy: 60% to 5.74% | Hurt |
| Fine-tune on HotpotQA (ablation) | Flan-T5-XL · FailureSensorIQ SC-MCQA | accuracy: 36% to 29% | Hurt |
| Perturb the question content and format (ablation) | o1 · FailureSensorIQ SC-MCQA ComplexPert | accuracy: 60.4% to 44.2% | Hurt |
| Perturb the question content and format (ablation) | Llama 4 Scout · FailureSensorIQ SC-MCQA ComplexPert | accuracy: 54.0% to 24.1% | Hurt |
| Increase the number of selected options under perturbation (ablation) | o1 · FailureSensorIQ SC-MCQA | additional incorrect options: 11.20% to 23.6% | Hurt |
Surprising findings
- Tool-augmented ReAct retrieval from Wikipedia and arXiv generally reduced accuracy rather than improving it.
- Fine-tuning Flan-T5-XL on HotpotQA lowered FailureSensorIQ accuracy from 36% to 29%.
Limitations
- Focuses on static knowledge and does not model temporal sensor-failure dynamics.
- Performance is affected by uneven online knowledge availability across assets.
- Human expert agreement is only moderate (Cohen's kappa 0.462).
Experimental setup
Datasets: ISO document failure-mode/sensor mapping tables HotpotQA
Benchmarks: FailureSensorIQ
Metrics: accuracy consistency-based accuracy set size coverage rate uncertainty-adjusted accuracy performance drop ratio
Baselines: o1 o3-mini gpt-4.1 deepseek-r1 llama-4-scout-17b-16e mistral-large-instruct-2407 flan-t5-xl
Builds on: TruthfulQA PertEval ReAct Plan-and-Solve prompting
Closest papers by meaning
- CofCA: A STEP-WISE Counterfactual Multi-hop QA benchmark (ICLR 2025 · similarity 0.86)
- Evaluating LLM Reasoning in the Operations Research Domain with ORQA (AAAI 2025 · similarity 0.85)
- AbstentionBench: Reasoning LLMs Fail on Unanswerable Questions (NeurIPS 2025 · similarity 0.85)
- Explore What LLM Does Not Know in Complex Question Answering (AAAI 2025 · similarity 0.85)
- MDK12-Bench: A Multi-Discipline Benchmark for Evaluating Reasoning in Multimodal Large Language Models (AAAI 2026 · similarity 0.84)
- Diagnosing and Addressing Pitfalls in KG-RAG Datasets: Toward More Reliable Benchmarking (NeurIPS 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.