Research map / Vision-language and multimodal / CLIP and zero-shot
EEG-Language Pretraining for Highly Label-Efficient Clinical Phenotyping
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces EEG-language models (ELMs) pretrained on 15k TUEG EEGs paired with clinical reports, using sub-unit alignment with text segmentation, EEG cropping, and a multiple-instance MIL-InfoNCE loss, improving label-efficient pathology detection and enabling zero-shot classification and retrieval.
Problem
Clinical EEG pathology detection suffers from scarce labels, and EEG-report pairs are long, heterogeneous, and temporally unaligned, so standard vision-language contrastive pretraining does not directly apply.
Approach
Align multiple non-overlapping EEG crops with multiple segmented report sections via a MIL-extended InfoNCE (MIL-NCE style, bidirectional positives), with sampled text clusters and a frozen MedCPT text encoder, so the EEG encoder learns from weak, many-to-many EEG-text correspondences.
Key results
- ELM-MIL improves TUAB linear probing at 1% labels by up to 8.7% balanced accuracy and 9.7% AUROC over EEG-only pretraining.
- ELM-MIL e,l achieves zero-shot TUAB balanced accuracy of 84.31% and AUROC of 91.56%.
- ELM-MIL reaches 84.42% balanced accuracy and 91.44% AUROC on TUAB at 80% labels, slightly above LaBraM-Huge (369M) in AUROC average with a 0.93M encoder.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use multimodal ELM-MIL rather than EEG-only self-supervised pretraining | EEG encoders · TUAB, 1% labeled training data | balanced accuracy: 74.99 -> 83.71 | Helped |
| Use multimodal ELM-MIL rather than EEG-only self-supervised pretraining | EEG encoders · NMT, 1% labeled training data | AUROC: 65.72 -> 69.49 | Helped |
| Use bidirectional alignment with MIL-InfoNCE instead of the tested alternative positive-sample aggregations (ablation) | ELM-MIL e,l · EEG retrieval, Top-10 | accuracy: 3.9, 8.3, 24.7 -> 27.1 | Helped |
| Increase positive EEG samples from 2 to 32 (ablation) | ELM-MIL e,l · EEG retrieval, Top-10 | accuracy: 19.8 -> 27.1 | Helped |
| Increase positive text samples from 2 to 8 (ablation) | ELM-MIL e,l · TUAB linear probe, 100% | balanced accuracy: 85.8 -> 87.1 | Helped |
| Train ELM e,l with correctly paired reports rather than shuffled reports (ablation) | ELM e,l · TUAB pathology detection with few annotations | accuracy | Helped |
| Shuffle reports between patients during pretraining (ablation) | ELM e,l · TUAB linear probing | accuracy: 73.70%, 81.04%, 83.69% | Helped |
| Use ELM e,l rather than ELM l, without a text projection head (ablation) | EEG-language pretraining · Report retrieval | Top-K accuracy | Helped |
| Use MedCPT rather than BiomedBERT as the language encoder (ablation) | ELM e,l · TUAB zero-shot | balanced accuracy: 78.61{ $\pm$2.90} -> 82.58{ $\pm$0.25} | Helped |
| Use MedCPT rather than Bio-ClinicalBERT as the language encoder (ablation) | ELM e,l · TUAB zero-shot | AUROC: 87.33{ $\pm$0.68} -> 88.37{ $\pm$0.39} | Helped |
Surprising findings
- Pretraining on shuffled (semantically irrelevant) reports still boosts pathology detection over an untrained encoder and nearly matches EEG-only pretraining, suggesting sub-unit alignment itself encodes between-subject information.
- Zero-shot performance requires a text projector, clinician interpretation text, and bidirectional alignment; ELM-l models trained on clinical history or interpretation alone performed poorly in some settings.
Limitations
- Limited publicly available paired EEG-report data restricts pretraining scale.
- Model size scaling was not studied due to computational constraints.
- Underperforms on artifact and eye-movement TUEV classes, possibly because related text was segmented out.
Experimental setup
Datasets: TUEG TUAB NMT TUSZ TUEV TUH EEG Epilepsy Corpus
Benchmarks: TUAB NMT TUSZ TUEV
Metrics: balanced accuracy AUROC top-k retrieval accuracy F1
Baselines: BYOL VICReg ContraWR Relative Positioning Temporal Shuffling CPC LaBraM BIOT CEReBrO SPaRCNet
Builds on: CLIP MIL-NCE M-FLAG VICReg BYOL
Closest papers by meaning
- Neural Signals Generate Clinical Notes in the Wild (NeurIPS 2026 · similarity 0.88)
- CLEF: EEG Foundation Models for Learning Clinical Semantics (NeurIPS 2026 · similarity 0.87)
- Large Brain Model for Learning Generic Representations with Tremendous EEG Data in BCI (ICLR 2024 · similarity 0.87)
- NeuroAtlas: Benchmarking Foundation Models for Clinical EEG and Brain-Computer Interfaces (NeurIPS 2026 · similarity 0.87)
- LEAF: Language-EEG Aligned Foundation Model for Brain-Computer Interfaces (NeurIPS 2026 · similarity 0.86)
- Standing on the Shoulders of Giants: Rethinking EEG Foundation Model Pretraining via Multi-Teacher Distillation (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.