Research map / Vision-language and multimodal / Cross-modal learning

CORTEG: Foundation Models Enable Cross-Modality Representation Transfer from Scalp to Intracranial Brain Recordings

NeurIPS 2026 · Vision-language and multimodal › Cross-modal learning · approach: Medical multimodal fusion · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Adapts the scalp-EEG foundation model ST-EEGFormer to intracranial ECoG decoding via a KNNSoftFourier electrode-coordinate adapter, dual-stream LFS/HGA tokenizer, and LoRA with leave-one-subject-out fine-tuning.

Problem

ECoG decoders are small and subject-specific because per-patient data is scarce, and it is unclear whether large scalp-EEG foundation models can transfer to ECoG despite mismatches in electrode geometry, spectrum, and data availability.

Approach

Map subject-specific ECoG electrode coordinates into a frozen pretrained EEG channel-embedding codebook using a soft kNN-kernel attention branch plus a zero-initialized Fourier residual branch, feed a dual-stream (low-frequency and high-gamma) tokenizer into the EEG ViT, adapt the last layers with LoRA, and calibrate a new patient via leave-one-subject-out fine-tuning of the adapter, LoRA, and hea

Key results

• Pooled CORTEG reaches mean Pearson r=0.554 on the finger task vs DeepFingerNet 0.542 and HiLoFuseNet 0.534, not statistically significant (p=0.65 vs DeepFingerNet).
• On the audio envelope task, pooled CORTEG reaches r=0.339 vs CNN-LSTM 0.261, significant after Bonferroni correction.
• LOO-FT with LoRA reaches r=0.551 (finger) and 0.331 (audio), close to pooled training, and calibrates a new patient in 10-30 minutes on a single GPU with 297K trainable parameters.

Surprising findings

• Scaling the backbone from Small to Base or Large did not improve performance significantly (Base 0.583 vs Small 0.577 in HPC sweep; Large slightly lower).
• Random-initialized backbone reaches 0.510 on finger, close to pretrained 0.554, while pretraining matters much more on audio (0.156 vs 0.339).

Limitations

• Finger-task improvement over strongest baselines is not statistically significant on n=9 subjects.
• Shared MNI-space electrode adapter cannot fully exploit subject-specific anatomy or functional organization.
• No cross-task transfer evaluated and no scaling law observed with backbone size.

Experimental setup

Datasets: Stanford fingerflex ECoG private audio-envelope ECoG scalp EEG pretraining corpus (ST-EEGFormer)

Benchmarks: Stanford fingerflex private audio envelope ECoG

Metrics: Pearson correlation paired Wilcoxon test participation ratio

Baselines: Ridge regression PLS HOPLS LSTM CNN-LSTM DeepFingerNet HiLoFuseNet LaBraM CBraMod MantisV2

Builds on: ST-EEGFormer LoRA Fourier feature positional encoding Nadaraya-Watson kernel regression

Closest papers by meaning

• CSBrain: A Cross-scale Spatiotemporal Brain Foundation Model for EEG Decoding (NeurIPS 2025 · similarity 0.87)
• Neural decoding from stereotactic EEG: accounting for electrode variability across subjects (NeurIPS 2024 · similarity 0.87)
• NeuroAtlas: Benchmarking Foundation Models for Clinical EEG and Brain-Computer Interfaces (NeurIPS 2026 · similarity 0.87)
• NeurIPT: Foundation Model for Neural Interfaces (NeurIPS 2025 · similarity 0.86)
• CBraMod: A Criss-Cross Brain Foundation Model for EEG Decoding (ICLR 2025 · similarity 0.86)
• What do EEG Foundation Models Capture from Human Brain Signals? (NeurIPS 2026 · similarity 0.86)