Research map / Label-efficient and robust learning / Self-supervised and contrastive

MOTOR: A Time-to-Event Foundation Model For Structured Medical Records

ICLR 2024 · Label-efficient and robust learning › Self-supervised and contrastive · approach: Self-supervised neuroimaging representations · 10 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Pretrains a 143M-parameter Transformer piecewise-exponential time-to-event foundation model (MOTOR) on up to 55M patient records via a self-supervised TTE objective over thousands of tasks, then adapts it to 19 downstream TTE tasks.

Problem

Time-to-event prediction on structured medical records is label-hungry and heavily censored, and prior pretraining for EHR focused on next-code objectives that do not capture long-horizon event timing; large TTE foundation models were also not released.

Approach

Pretrain a causal, local-attention Transformer with a piecewise exponential likelihood over thousands of time-to-next-code tasks simultaneously, using a low-rank time-independent task embedding and sparse/fused loss computation to scale, then adapt via linear probe or full finetuning.

Key results

• MOTOR-Finetune improves time-dependent C statistic over the best baseline on code-based tasks, e.g., Celiac 0.802 vs 0.729 (RSF) on EHR-OMOP.
• TTE pretraining outperforms next-code pretraining on EHR-OMOP, e.g., HA 0.887 vs 0.862 and Stroke 0.875 vs 0.857.
• MOTOR-Transfer Probe on MIMIC-IV outperforms best baseline on all six tasks, e.g., NAFLD 0.802 vs 0.736 and Cancer 0.828 vs 0.748.

Measured effects reported in the paper

Surprising findings

• MOTOR-Scratch underperforms baselines such as RSF on several EHR-OMOP tasks, indicating the architecture alone is not the source of gains without pretraining.
• Pretraining on a smaller data subset degrades MOTOR rapidly, while adaptation with fewer labels is robust.

Limitations

• Pretraining is expensive, requiring up to two GPU weeks for MERATIVE.
• Evaluation uses subsampled censored cases, introducing censoring bias in target tasks.
• Competing risks are handled by censoring at death, which may not suit all settings.

Experimental setup

Datasets: EHR-OMOP MERATIVE MIMIC-IV

Benchmarks: EHR-OMOP code-based tasks MERATIVE code-based tasks EHR-OMOP text-based radiology tasks MIMIC-IV transfer tasks out-of-time EHR-OMOP

Metrics: time-dependent C statistic Harrell's C-index Nam-D'Agostino calibration Integrated Brier Score

Baselines: Cox PH DeepSurv DeepHit DSM RSF MOTOR-Scratch next code pretraining

Builds on: PEANN BEHRT rotary position embedding

Closest papers by meaning

• One Loss to Rule Them All: Marked Time-to-Event for Structured EHR Foundation Models (NeurIPS 2026 · similarity 0.88)
• XTSFormer: Cross-Temporal-Scale Transformer for Irregular-Time Event Prediction in Clinical Applications (AAAI 2025 · similarity 0.86)
• Context Clues: Evaluating Long Context Models for Clinical Prediction Tasks on EHR Data (ICLR 2025 · similarity 0.86)
• Time-to-Event Pretraining for 3D Medical Imaging (ICLR 2025 · similarity 0.85)
• Risk Horizons: Structured Hypothesis Spaces for Longitudinal Clinical Prediction (NeurIPS 2026 · similarity 0.85)
• Integrating Sequence and Image Modeling in Irregular Medical Time Series Through Self-Supervised Learning (AAAI 2025 · similarity 0.84)