Research map / Large language models / Language, safety and interpretability
Structured Document Generation for Industrial Equipment
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Describes an LLM-based application that generates FMEA structured documents for industrial equipment using supervised in-context learning, structured generation, and counter-based entity ensembling, with SME-supervised UI and new SSEE metrics.
Problem
Creating Failure Modes and Effects Analyses (FMEAs) for industrial equipment is expert-intensive and costly, and existing text metrics (ROUGE, BLEU, perplexity) do not evaluate document structure or entity-level semantics.
Approach
Combines supervised dynamic few-shot prompting over a library of ~700 expert FMEAs, structured multi-step prompting with parsers and validation, and a counter-based entity ensembling (CBEE) that merges entity lists from multiple LLMs/prompts, plus entity-level SSEE metrics and an SME-facing UI.
Key results
- CBEE ensemble with count threshold 1 raised SSEE recall to 0.669 versus 0.605 for the best single response (Llama 3, 15-shot DFSP).
- CBEE with count threshold 5 reached SSEE precision of 0.91, at the cost of recall (0.309).
- An SME generated 100 new FMEAs, averaging about four hours per FMEA, versus several person-days traditionally.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add DFSP in-context examples (ablation) | Flan-UL2 · FMEA component generation, test split | SSEE Recall: 0.047 -> 0.470 | Helped |
| Add DFSP in-context examples (ablation) | Llama 3 · FMEA component generation, test split | SSEE Recall: 0.176 -> 0.605 | Helped |
| Add DFSP in-context examples (ablation) | Mixtral 8x7B · FMEA component generation, test split | SSEE Recall: 0.135 -> 0.513 | Helped |
| Increase DFSP examples from 5 to 15 (ablation) | Individual models · FMEA component generation, test split | SSEE Precision | Mixed |
| Vary DFSP shot ordering (ablation) | Individual models · FMEA component generation, test split | SSEE F1 | No clear effect |
| Add CBEE ensembling with count threshold 1 | Multiple model and prompt outputs · FMEA component generation, test split | SSEE Recall: 0.605 -> 0.669 | Helped |
| Add CBEE ensembling with count threshold 5 | Multiple model and prompt outputs · FMEA component generation, test split | SSEE Precision: 0.910 | Helped |
| Increase CBEE count threshold from 2 to 3 (ablation) | Many-model Best 3 ensemble · FMEA component generation, test split | SSEE F1: 0.604 -> 0.515 | Hurt |
| Use the Best 5 rather than Best 3 outputs (ablation) | Many-model CBEE ensemble, threshold 2 · FMEA component generation, test split | SSEE Recall: 0.558 -> 0.610 | Helped |
| Use the Best 5 rather than Best 3 outputs (ablation) | Many-model CBEE ensemble, threshold 2 · FMEA component generation, test split | SSEE Precision: 0.658 -> 0.584 | Hurt |
Surprising findings
- Including more shots (15 vs 5) improved most metrics but not SSEE precision, possibly due to irrelevant shots the models fail to filter.
Limitations
- Evaluated on a single asset-component generation task with a proprietary database, limiting generalizability.
- Generated FMEAs still take several person-hours each to review and correct.
- Precision-recall trade-off in CBEE means high-precision settings sacrifice substantial recall.
Experimental setup
Datasets: proprietary FMEA database (714 SME-created FMEAs)
Benchmarks: proprietary FMEA component generation test split (72 examples)
Metrics: SSEE Recall SSEE Precision SSEE F1 ROUGE-1
Baselines: Zero-shot prompting Flan-UL2 DFSP Llama 3 DFSP Mixtral 8x7B DFSP
Builds on: dynamic few-shot prompting LLM-BLENDER self-consistency all-mpnet-base-v2
Closest papers by meaning
- AAAR-1.0: Assessing AI’s Potential to Assist Research (ICML 2025 · similarity 0.80)
- IDEA-Bench: How Far are Generative Models from Professional Designing? (CVPR 2025 · similarity 0.80)
- AnyDoc: Enhancing Document Generation via Large-Scale HTML/CSS Data Synthesis and Height-Aware Reinforcement Optimization (CVPR 2026 · similarity 0.80)
- IndustryCode: A Benchmark for Industry Code Generation (NeurIPS 2026 · similarity 0.80)
- LongGenBench: Benchmarking Long-Form Generation in Long Context LLMs (ICLR 2025 · similarity 0.80)
- Raidar: geneRative AI Detection viA Rewriting (ICLR 2024 · similarity 0.80)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.