Research map / Large language models / Instruction and fine-tuning
NeuroLM: A Universal Multi-task Foundation Model for Bridging the Gap between Language and EEG Signals
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes NeuroLM, an EEG foundation model that tokenizes EEG with a text-aligned VQ neural tokenizer, pre-trains a GPT-2-based LLM with multi-channel autoregression on ~25,000 hours of EEG, and performs multi-task instruction tuning across six EEG datasets.
Problem
Existing EEG foundation models need separate full fine-tuning for each downstream task, limiting versatility and wasting resources; aligning EEG with language for multi-task LLM-based EEG processing is unsolved.
Approach
Treat EEG as a foreign language: a VQ neural tokenizer trained with temporal-frequency reconstruction and adversarial domain alignment to text space produces discrete EEG tokens, which a causal LLM models via multi-channel autoregression and instruction-tunes jointly across tasks.
Key results
- On TUAB, NeuroLM-XL (multi-task) reaches 0.7969 balanced accuracy and 0.7884 AUROC, below single-task LaBraM-Base at 0.8140 and 0.9022.
- On HMC, NeuroLM-B (multi-task) reaches 0.6737 balanced accuracy, close to single-task LaBraM-Base at 0.7286.
- On TUSL, NeuroLM-XL reaches 0.6845 balanced accuracy versus 0.7625 for LaBraM-Base; the paper states NeuroLM is still a few steps behind LaBraM.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase model size from NeuroLM-B to NeuroLM-XL (ablation) | NeuroLM · TUAB, TUEV, SEED, HMC, Workload, and TUSL | Balanced Acc.: TUAB 0.7826$\pm$0.0065 -> 0.7969$\pm$0.0091; TUEV 0.4560$\pm$0.0048 -> 0.4679$\pm$0.0356; SEED 0.5554$\pm$0.0075 -> 0.60 | Mixed |
| Add multi-channel autoregressive pre-training (ablation) | NeuroLM · Downstream tasks | Performance | Helped |
| Remove EEG-text embedding space alignment (ablation) | NeuroLM neural tokenizer · Downstream tasks | Performance | Hurt |
| Shuffle multiple-choice option order during instruction tuning (ablation) | NeuroLM · TUEV and HMC | Performance | No clear effect |
| Shuffle multiple-choice option order during instruction tuning (ablation) | NeuroLM · TUSL | Performance | Hurt |
| Reconstruct both temporal and frequency domains rather than only one domain (ablation) | NeuroLM neural tokenizer · TUAB, TUEV, SEED, HMC, Workload, and TUSL | Performance | Mixed |
| Use 20 instead of 5 pre-training epochs (ablation) | NeuroLM · TUAB | Balanced Acc.: 0.7763$\pm$0.0037 -> 0.7826$\pm$0.0065 | Helped |
| Use 20 instead of 10 pre-training epochs (ablation) | NeuroLM · TUEV | Balanced Acc.: 0.4693$\pm$0.0175 -> 0.4560$\pm$0.0048 | Hurt |
| Use 20 instead of 5 pre-training epochs (ablation) | NeuroLM · SEED | Balanced Acc.: 0.5641$\pm$0.0103 -> 0.5554$\pm$0.0075 | Hurt |
| Use 20 instead of 5 pre-training epochs (ablation) | NeuroLM · HMC | Balanced Acc.: 0.6956$\pm$0.0136 -> 0.6737$\pm$0.0050 | Hurt |
| Use 20 instead of 10 pre-training epochs (ablation) | NeuroLM · Workload | Balanced Acc.: 0.6540$\pm$0.0192 -> 0.6172$\pm$0.0113 | Hurt |
| Use 20 instead of 5 pre-training epochs (ablation) | NeuroLM · TUSL | Balanced Acc.: 0.5342$\pm$0.0235 -> 0.6734$\pm$0.0436 | Helped |
Surprising findings
- Without EEG-text space alignment, the model outputs random words instead of answer options, yielding near-zero scores on most downstream metrics.
- Larger NeuroLM variants overfit more; performance degraded on HMC for the bigger models.
Limitations
- Still lags behind single-task state-of-the-art LaBraM on most datasets and is sensitive to hyperparameters.
- Only coarse space-wise EEG-text alignment is used due to scarce EEG-text pairs.
- Uses GPT-2 as a relatively small base LLM; larger models overfit and show unstable performance on small datasets such as TUSL.
Experimental setup
Datasets: TUEG SEED series BCI Competition IV-1 Emobrain Grasp and Lift Inria BCI Motor Movement/Imagery Raw EEG Data Resting State Siena Scalp EEG Database SPIS Resting State Target Versus Non-Target
Benchmarks: TUAB TUEV SEED HMC Workload TUSL
Metrics: Balanced Accuracy AUC-PR AUROC Cohen's Kappa Weighted F1
Baselines: SPaRCNet ContraWR CNN-Transformer FFCL ST-Transformer BIOT LaBraM-Base
Builds on: LaBraM BIOT GPT-2 VQ-VAE LLaVA
Closest papers by meaning
- BrainLM: A foundation model for brain activity recordings (ICLR 2024 · similarity 0.89)
- LEAF: Language-EEG Aligned Foundation Model for Brain-Computer Interfaces (NeurIPS 2026 · similarity 0.89)
- Neural Signals Generate Clinical Notes in the Wild (NeurIPS 2026 · similarity 0.88)
- EEG Benchmarking Needs a Task Specification Layer: NeuroDoc for Rulebook-Guided, Executable Benchmark Construction (NeurIPS 2026 · similarity 0.88)
- NeurIPT: Foundation Model for Neural Interfaces (NeurIPS 2025 · similarity 0.88)
- NeuroAtlas: Benchmarking Foundation Models for Clinical EEG and Brain-Computer Interfaces (NeurIPS 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.