Research map / Label-efficient and robust learning / Recognition and long-tail
Yet Another ICU Benchmark: A Flexible Multi-Center Framework for Clinical ML
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces YAIB, a modular, reproducible multi-dataset benchmark framework for ICU clinical prediction tasks, with baselines across MIMIC-III/IV, eICU, HiRID, and AmsterdamUMCdb and analysis of how cohort and preprocessing choices shift performance.
Problem
Clinical ICU prediction models are hard to compare and reproduce because datasets, cohort definitions, preprocessing and training setups vary, and existing benchmarks are hard-coded for single datasets and fixed tasks.
Approach
An end-to-end, modular framework that harmonizes multiple open ICU datasets (via ricu concepts), defines cohorts and prediction tasks in a dataset-independent, configurable way, and trains/evaluates ML and DL models through gin-config files, so that task or preprocessing variations are transparent and reproducible.
Key results
- On the mortality task, GRU/TCN/TF reach about 84-86 AUROC on AmsterdamUMCdb, HiRID, eICU and MIMIC-IV, with LGBM and deep models highly comparable.
- Changing mortality exclusion criteria (>24h vs >30h) on HiRID shifted AUROC from 84.9 to 88.2 for the transformer, showing cohort definition strongly affects performance.
- Removing dynamic feature generation lowered LR AUROC on MIMIC-IV from 86.1 to 80.2 and LGBM from 87.7 to 85.9.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add dynamic feature generation (ablation) | Classical ML models for ICU mortality · ICU mortality, AUMC, HiRID, eICU, and MIMIC-IV | AUROC: LR: 82.2±0.5 -> 83.7±0.6 (AUMC); 81.8±0.7 -> 84.0±0.3 (HiRID); 81.1±0.1 -> 84.8±0.2 (eICU); 80.2±0.3 -> 86.1±0.1 (MIMIC- | Helped |
| Add static features (ablation) | ICU mortality models · ICU mortality, AUMC, HiRID, eICU, and MIMIC-IV | AUROC: LR with/without static: 83.7±0.6 / 82.9±0.6; 84.0±0.3 / 82.8±0.7; 84.8±0.2 / 84.3±0.2; 86.1±0.1 / 84.3±0.3 | Helped |
| Allow ICU stays of 24–30 hours rather than excluding them (ablation) | ICU mortality cohort on HiRID · HiRID | AUROC: LR: 84.0±0.3 -> 87.2±0.4; LGBM: 84.5±0.3 -> 87.9±0.5; GRU: 84.8±0.2 -> 88.2±0.3; TCN: 84.6±0.7 -> 87.8±0.2; TF: 84.9±0.7 | Helped |
| Use the paper's clinically actionable sepsis definition requiring antibiotics continuously for ≥3 days (ablation) | Sepsis prediction models · Sepsis, MIMIC-IV | AUPRC: LGBM: 10.5±0.0 -> 4.3±0.0; GRU: 17.7±0.0 -> 6.1±0.0 | Hurt |
| Replace the paper's sepsis definition with the alternative definition (ablation) | Sepsis prediction models · Sepsis, MIMIC-IV | AUROC: GRU: 79.2±0.1 -> 80.9±0.0 -> 89.2±0.0 | Mixed |
| Use GRU instead of classical ML for AKI prediction | ICU prediction models · AKI across AUMC, HiRID, eICU, and MIMIC-IV | AUROC: GRU: 90.6±0.3, 82.2±0.2, 90.9±0.0, 90.7±0.1; best classical: 85.8±0.3, 80.2±0.2, 84.6±0.1, 83.8±0.1 | Helped |
| Fine-tune an eICU GRU on HiRID samples instead of training from scratch | GRU for ICU mortality · HiRID | AUROC | Helped |
Surprising findings
- Cohort definition choices (exclusion criteria, sepsis definition) affected performance as much as or more than model class.
- Pooled-dataset and eICU-pretrained models transferred reasonably well, but HiRID transfer from other datasets was weaker.
Limitations
- Restricted to ICU settings and currently to 52 shared features; medications, comorbidities and notes are not included.
- Specialized settings such as federated or reinforcement learning need custom code.
- Full datasets require credentialed access; only demo cohorts are freely available.
Experimental setup
Datasets: MIMIC-III MIMIC-IV eICU HiRID AmsterdamUMCdb MIMIC-III demo eICU demo
Benchmarks: YAIB ICU mortality task YAIB AKI task YAIB sepsis task YAIB kidney function task YAIB length of stay task
Metrics: AUROC AUPRC MAE
Baselines: LR EN LGBM GRU LSTM TCN Transformer Clairvoyance TemporAI PyHealth HiRID benchmark
Builds on: ricu gin-config PyTorch Lightning scikit-optimize
Closest papers by meaning
- HiRID-ICU-Benchmark --- A Comprehensive Machine Learning Benchmark on High-resolution ICU Data (NeurIPS 2021 · similarity 0.91)
- Predicting ICU Mortality Risk by Grouping Temporal Trends from a Multivariate Panel of Physiologic Measurements (AAAI 2016 · similarity 0.88)
- GARLIC: Graph Attention-based Relational Learning of Multivariate Time Series in Intensive Care (ICLR 2026 · similarity 0.87)
- ICU Mortality Prediction: A Classification Algorithm for Imbalanced Datasets (AAAI 2017 · similarity 0.87)
- CLIMB: Data Foundations for Large Scale Multimodal Clinical Foundation Models (ICML 2025 · similarity 0.84)
- Distribution-Free Uncertainty Quantification in Mechanical Ventilation Treatment: A Conformal Deep Q-Learning Framework (AAAI 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.