Research map / Vision-language and multimodal / Cross-modal learning

Brain-OF: An Omnifunctional Foundation Model for fMRI, EEG and MEG

NeurIPS 2026 · Vision-language and multimodal › Cross-modal learning · approach: Medical multimodal fusion · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Brain-OF jointly pretrains a 47M–1.7B-parameter transformer on ~37-40 public fMRI, EEG and MEG datasets using an any-resolution signal sampler, DINT attention, a sparse MoE, and a masked temporal-frequency reconstruction objective.

Problem

Existing brain foundation models are confined to a single neuroimaging modality, limiting data scale and exploitation of complementary fMRI, EEG and MEG information, and are hampered by semantic heterogeneity and resolution discrepancies across modalities.

Approach

Project variable-resolution fMRI/EEG/MEG patch sequences into a fixed set of latent tokens via Perceiver-style cross-attention (ARNESS), process them with DINT attention and a shared+routed sparse MoE, and pretrain by reconstructing masked signals in both time and frequency domains (MTFM).

Key results

• Brain-OF Huge reaches 26.36 Kappa on SEED-V, 74.87 BAC on CHB-MIT and 82.87 BAC on TUAB, ranking first in 7 of 9 tasks.
• On CamCAN MEG brain-age regression, MAE drops from 8.99 (Base) to 7.87 (Huge), the best among compared methods.
• Multimodal fusion improves over unimodal in CamCAN (fMRI+MEG MAE 8.78 with Large vs 8.86 MEG-only) and LEMON (fMRI+EEG BAC 78.32% vs 78.04% EEG-only with Large).

Surprising findings

• Removing EEG from pretraining causes a larger drop on ADHD-200 fMRI than removing fMRI itself, suggesting cross-modal data can matter more than within-modality data.
• Single-modality foundation models (LaBraM, BrainHarmonix-F) are sometimes outperformed by task-specific models on other modalities, indicating negative transfer.

Limitations

• Multimodal fusion gains over the best unimodal setting are small in several cases (e.g., LEMON Large 78.32 vs 78.04), with no significance-level reporting beyond standard deviations.
• Some pretraining datasets are noted as continuously expanding or have heterogeneous preprocessing, and downstream results rely on five seeds on a limited set of tasks.

Experimental setup

Datasets: FACED PhysioNet-MI PREDICT REEG-BACA SEED series TDBRAIN TUEG WBCIC-MI ABIDE-I ABIDE-II ABRIM AOMIC

Benchmarks: SEED-V CHB-MIT TUAB ADNI ADHD-200 CamCAN LEMON

Metrics: Balanced accuracy Cohen's Kappa MAE

Baselines: EEGNet EEGConformer ContraWR CNN-Transformer FFCL ST-Transformer BrainNetTF BrainNetCNN LaBraM CBraMod BrainHarmonix-F Brain-JEPA

Builds on: Differential Attention Perceiver LaBraM CBraMod Brain-JEPA

Closest papers by meaning

• BrainOmni: A Brain Foundation Model for Unified EEG and MEG Signals (NeurIPS 2025 · similarity 0.91)
• BrainLM: A foundation model for brain activity recordings (ICLR 2024 · similarity 0.90)
• Brain decoding: toward real-time reconstruction of visual perception (ICLR 2024 · similarity 0.88)
• Standing on the Shoulders of Giants: Rethinking EEG Foundation Model Pretraining via Multi-Teacher Distillation (NeurIPS 2026 · similarity 0.88)
• CSBrain: A Cross-scale Spatiotemporal Brain Foundation Model for EEG Decoding (NeurIPS 2025 · similarity 0.88)
• MIRAGE: Adaptive Multimodal Gating for Whole-Brain fMRI Encoding (NeurIPS 2026 · similarity 0.88)