Research map / Label-efficient and robust learning / Self-supervised and contrastive

SMART: Towards Pre-trained Missing-Aware Model for Patient Health Status Prediction

NeurIPS 2024 · Label-efficient and robust learning › Self-supervised and contrastive · approach: Self-supervised neuroimaging representations · 5 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes SMART, a self-supervised missing-aware transformer that uses mask-biased temporal and variable attention and pre-trains by reconstructing masked patient representations in latent space (EMA target encoder), then fine-tunes for EHR prediction tasks.

Problem

Missing values in sparse EHR time series cause spurious correlations and poor patient health status predictions; existing input-space imputation can add noise and distract from semantic information.

Approach

Encode missingness directly through mask-based attention biases (temporal and variable) and perform self-supervised pre-training that reconstructs masked visit representations in latent space using an EMA target encoder, with a learnable CLS vector bridging pre-training and fine-tuning.

Key results

• SMART reaches 53.84% AUPRC on Cardiology and 81.67% AUPRC on Sepsis, outperforming the best baseline on the binary tasks by an average of 3.80% AUPRC and 5.15% F1.
• On MIMIC-III decompensation SMART achieves 71.26% AUPRC versus 68.21% for Warpformer, the next best.
• Replacing latent-space reconstruction with input-space imputation pre-training lowers Cardiology AUPRC from 53.84 to 52.77.

Measured effects reported in the paper

Surprising findings

• Basic GRU and Transformer baselines are competitive on some tasks but Transformer collapses on Sepsis (22.56% AUPRC).
• Some strong baselines (StageNet, PrimeNet) perform surprisingly poorly on Sepsis despite good results on other datasets.

Limitations

• Only ICU multivariate time-series data is used; other EHR modalities (codes, notes, imaging) are not covered.
• Temporal attention complexity is O(T^2), limiting very long sequences.
• No interpretability analysis is provided.

Experimental setup

Datasets: Cardiology (PhysioNet 2012) Sepsis (PhysioNet 2019) MIMIC-III

Benchmarks: Cardiology Sepsis MIMIC-III in-hospital mortality MIMIC-III decompensation MIMIC-III phenotyping MIMIC-III length of stay

Metrics: AUPRC F1 Score ma-ROC mi-ROC AUROC min(Se, P+)

Baselines: AdaCare StageNet ConCare GRASP SAFARI PPN RainDrop Warpformer PrimeNet GRU Transformer

Builds on: BERT PrimeNet Transformer exponential moving average teacher

Closest papers by meaning

• GRASP: Generic Framework for Health Status Representation Learning Based on Incorporating Knowledge from Similar Patients (AAAI 2021 · similarity 0.85)
• Integrating Sequence and Image Modeling in Irregular Medical Time Series Through Self-Supervised Learning (AAAI 2025 · similarity 0.84)
• Contrastive Learning for Clinical Outcome Prediction with Partial Data Sources (ICML 2024 · similarity 0.83)
• DearLLM: Enhancing Personalized Healthcare via Large Language Models-Deduced Feature Correlations (AAAI 2025 · similarity 0.83)
• Learning What Not to Impute: An Uncertainty-Aware Diffusion Framework for Meaningful Missingness (NeurIPS 2026 · similarity 0.83)
• Distribution Shift in Missing Data Imputation: A Risk-Based Perspective and Importance-Weighted Correction under MAR (NeurIPS 2026 · similarity 0.83)