Research map / Vision-language and multimodal / Human and embodied modeling

LEAF: Language-EEG Aligned Foundation Model for Brain-Computer Interfaces

NeurIPS 2026 · Vision-language and multimodal › Human and embodied modeling · approach: fMRI visual decoding

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes LEAF, an EEG foundation model that pretrains with joint spectral-temporal reconstruction and aligns EEG to frozen-text task/label embeddings via an instruction-conditioned Q-Former, reporting best average balanced accuracy across five BCI paradigms on 16 datasets.

Problem

Existing EEG foundation models train on discrete task labels and do not exploit semantic structure of natural-language task instructions, nor jointly model spectral and multi-timescale temporal EEG dynamics.

Approach

A two-stage design: a spectral-temporal reconstruction pretraining (random frequency-band suppression plus bidirectional masked and causal transformer branches), followed by an Instruction-conditioned Q-Former that FiLM-modulates EEG tokens with a frozen sentence-embedding of the instruction and aligns query outputs to frozen label-name embeddings via cosine loss.

Key results

• LEAF reaches 64.14% overall balanced accuracy vs 61.97% for the next-best (CBraMod), with best average across MI, emotion, and other paradigms.
• LEAF achieves 71.36% balanced accuracy on the 'Other' group (SSVEP, covert speech, workload), exceeding the next-best method by over 4.7 points.
• Richer instructions improve direct-inference mean balanced accuracy from 0.4738 (no instruction) to 0.5004 (task and target instruction).

Surprising findings

• Misleading instructions reshape the learned EEG feature space toward the instruction's semantics rather than the ground-truth task.
• LEAF does not beat all baselines on every dataset; e.g., CBraMod and LaBraM are better on SHU-MI and PhysioNet-MI, and Conformer on OpenBMI-MI.

Limitations

• Results reported with mean and std over three seeds, but LEAF is not best on all datasets (e.g., SHU-MI, PhysioNet-MI, SEED valid for CBraMod).
• Text encoder is frozen and pre-computed; the semantic contribution depends on label-name choice and encoder quality.
• Direct-inference gains depend on informative instructions, so performance is sensitive to instruction wording.

Experimental setup

Datasets: Stieger2021 SEED-FRA SEED-GER SEED-SD SEED-Neg ChineseEEG Chisco LargeSpanish ThinkOutLoud BCIC-IV-2a OpenBMI-MI BCIC-Upperlimb

Benchmarks: BCIC-IV-2a OpenBMI-MI BCIC-Upperlimb SHU-MI HighGamma Cho2017 Shin2017A PhysioNet-MI FACED SEED SEED-IV SEED-V

Metrics: balanced accuracy Cohen's Kappa

Baselines: EEGNet TSception ST-Transformer Conformer BIOT EEGPT LaBraM CBraMod NeuroLM

Builds on: LaBraM CBraMod NeuroLM Q-Former FiLM

Closest papers by meaning

• MindAlign: Bridging EEG, Vision, and Language for Zero-Shot Visual Decoding (NeurIPS 2026 · similarity 0.89)
• NeuroLM: A Universal Multi-task Foundation Model for Bridging the Gap between Language and EEG Signals (ICLR 2025 · similarity 0.89)
• BrainOmni: A Brain Foundation Model for Unified EEG and MEG Signals (NeurIPS 2025 · similarity 0.88)
• SENSE: Semantic Neural Speech Synthesis from Brain Dynamics via Spatial Graph Encoding (NeurIPS 2026 · similarity 0.87)
• CSBrain: A Cross-scale Spatiotemporal Brain Foundation Model for EEG Decoding (NeurIPS 2025 · similarity 0.87)
• What Does the Brain See? Multiview Neural Representation to Demystify the Brain-Visual Alignment (NeurIPS 2026 · similarity 0.87)