Research map / Large language models / AI and society

PICID: A Modular Evaluation Infrastructure for Reproducible PHM Across Tasks and Domains

NeurIPS 2026 · Large language models › AI and society · approach: Benchmarking and reproducibility practice · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces PICID, a modular, protocol-centric evaluation library for PHM that executes data splits, preprocessing, target alignment, windowing, and metrics for fault detection, diagnostics, and prognostics, and benchmarks 13 models on 12 datasets.

Problem

PHM results are hard to reproduce and compare because protocol choices (splits, preprocessing, label alignment, windowing, metrics) are implicit or ad hoc, and existing resources do not enforce a shared, leakage-safe evaluation protocol across tasks and domains.

Approach

Treat the PHM evaluation protocol as an explicit executable artifact: formal operators for feature transform, target alignment, windowing, and partitioning, implemented as a YAML-configured modular library with train-only fitting and split-aware containers, so diagnostics and prognostics share identical data contracts and wrappers.

Key results

• Across 13 models and 12 datasets (150 model-dataset pairs), TabPFN and TabDPT achieve the top two average ranks, outperforming deep sequence models and transformers on average.
• Transformer models (Crossformer, TST, STF, PatchTST) perform competitively on several prognostics tasks but collapse to F1 of 17-40 on NC-D and MZVAV diagnostics.
• A 2022-2025 audit of 329 PHM Society and IJPHM papers found publicly available code in only 8 papers (2.4%) and data in 2 (0.6%).

Surprising findings

• Transformer family performs competitively on prognostics but near-random on most diagnostics tasks.
• Tabular foundation models (TabPFN, TabDPT) outperform deep sequence models and transformers on average across both task categories.

Limitations

• Does not resolve dataset quality, label noise, domain shift, or inconsistent failure definitions across PHM benchmarks.
• Benchmark suite limited in scale and does not capture many industrial settings or long-horizon deployment.
• Focuses on supervised offline diagnostics and prognostics; fault detection supported but omitted from main evaluation; continual, online, and uncertainty-aware settings out of scope.

Experimental setup

Datasets: N-CMAPSS-DS02 N-CMAPSS-P NB14 Unibo PHME20 XJTU-SY N-CMAPSS multi-source concepts HSF15 MZVAV

Benchmarks: NC-DS02 NC-P NB14 PHME20 Unibo XJTU-SY NC-D HSF15-A HSF15-C HSF15-P HSF15-V MZVAV

Metrics: MAE (normalized) F1 RMSE MAE PHM score accuracy macro-F1 AUROC AUPRC

Baselines: Linear regression Exponential regression MLP LSTM CNN-1D TiDE Timeseries Transformer Spacetimeformer Crossformer PatchTST XGBoost TabPFN

Builds on: sktime aeon PHM-Vibench ProgPy

Closest papers by meaning

• Benchmark Health Index: A Systematic Framework for Benchmarking the Benchmarks of LLMs (NeurIPS 2026 · similarity 0.82)
• Reliably detecting model failures in deployment without labels (NeurIPS 2025 · similarity 0.82)
• MD tree: a model-diagnostic tree grown on loss landscape (ICML 2024 · similarity 0.82)
• Specialized Foundation Models Struggle to Beat Supervised Baselines (ICLR 2025 · similarity 0.81)
• BearLLM: A Prior Knowledge-Enhanced Bearing Health Management Framework with Unified Vibration Signal Representation (AAAI 2025 · similarity 0.81)
• Statistically Valid Post-Deployment Monitoring Should Be Standard for AI-Based Digital Health (NeurIPS 2025 · similarity 0.81)