Research map / Federated learning and privacy / Privacy attacks and secure learning

An Investigation of Memorization Risk in Healthcare Foundation Models

NeurIPS 2025 · Federated learning and privacy › Privacy attacks and secure learning · approach: Encryption-based private learning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes a black-box, prompt-only suite of six tests (T1–T6) for assessing patient-level memorization risk in structured-EHR foundation models, and demonstrates it on the public EHRMamba2 model trained on MIMIC-IV.

Problem

Foundation models trained on structured EHR data may memorize patient information, creating privacy risks, but there is no established definition or practical evaluation of memorization in this setting; standard leakage metrics ignore clinical context and sensitivity.

Approach

A contextual, risk-based evaluation framework that distinguishes harmful patient-level memorization from benign population-level generalization, combining generative tests (trajectory similarity via time-weighted EMD over MedBERT embeddings, sensitive-diagnosis elicitation), embedding probing and membership inference, and perturbation and subgroup tests for identifying high-risk individuals.

Key results

• Prompting EHRMamba2 with more patient codes lowers trajectory EMD distance, and sensitive-diagnosis AUROC rises from about 0.55-0.62 (static) to about 0.72-0.75 (50 codes).
• Linear probing of frozen embeddings for sensitive diagnoses yields AUROC around 0.5, and membership inference shows no statistically significant separation between members and non-members.
• Rare-diagnosis prompts elicited sensitive codes such as sedative/anxiolytic dependence and mild manic episode, flagged as potential memorization concerns.

Measured effects reported in the paper

Surprising findings

• Samples with low trajectory distance do not necessarily carry higher risk; best-performing trajectories were short routine lab sequences.
• Positive sensitive-attribute predictions can reflect meaningful clinical reasoning (e.g., low CD8 count suggesting HIV) rather than memorization.

Limitations

• Benchmark is demonstrated on a single model (EHRMamba2) and one dataset (MIMIC-IV), limiting generality.
• Sensitive attributes are defined by an example ICD-10 categorization that may vary across settings.
• Tests are not exhaustive and the paper does not propose mitigation methods.

Experimental setup

Datasets: MIMIC-IV

Benchmarks: MIMIC-IV

Metrics: AUROC AUPRC precision recall earth mover's distance probability of generating sensitive token

Baselines: Static prompt baseline Random prompt setup

Builds on: EHRMamba MedBERT Earth Mover's Distance contextual integrity

Closest papers by meaning

• Quantifying and Analyzing Entity-Level Memorization in Large Language Models (AAAI 2024 · similarity 0.85)
• Context Clues: Evaluating Long Context Models for Clinical Prediction Tasks on EHR Data (ICLR 2025 · similarity 0.84)
• Collaborative Synthesis of Patient Records through Multi-Visit Health State Inference (AAAI 2024 · similarity 0.84)
• One Loss to Rule Them All: Marked Time-to-Event for Structured EHR Foundation Models (NeurIPS 2026 · similarity 0.84)
• Enhancing Small Medical Learners with Privacy-preserving Contextual Prompting (ICLR 2024 · similarity 0.84)
• IGAMT: Privacy-Preserving Electronic Health Record Synthesization with Heterogeneity and Irregularity (AAAI 2024 · similarity 0.83)