Research map / Large language models / AI and society

EEG Benchmarking Needs a Task Specification Layer: NeuroDoc for Rulebook-Guided, Executable Benchmark Construction

NeurIPS 2026 · Large language models › AI and society · approach: Benchmarking and reproducibility practice · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes a rulebook-guided EEG task specification language (task document plus executable kernel per dataset), with NeuroDoc (LLM-assisted drafting) and NeuroAudit (review UI), releases a 53-entry community-reviewed EEG benchmark corpus, and validates it by running four EEG foundation models on 18 datasets.

Problem

Public EEG datasets lack a shared task specification layer; task semantics (epoch derivation, labels, evidence, executable realization) are scattered across papers, code, and manual interpretation, hindering reusable multi-dataset EEG benchmarking.

Approach

Pair a human-readable task document with a synchronized executable task kernel per dataset, governed by a shared rulebook with machine-checkable constraints (evidence priority, raw vs. super meta, label routes, review states), plus LLM-assisted drafting (NeuroDoc) and a reviewer-facing audit UI (NeuroAudit).

Key results

• Corpus of 53 completed and reviewed entries with 245 task definitions (134 classical, 111 potential).
• Four backbones (BENDR, CBraMod, CodeBrain, LaBraM) ran across 266 runs under the shared layer, with a mean within-subject minus cross-subject macro-F1 gap of 12.6 points.
• NeuroDoc generation on 43 datasets: 39/43 documents generated, 0.0% placeholder DOI and missing-evidence rates, but only 47.6% kernel validation success and 36.1% generated-task redundancy.

Surprising findings

• NeuroDoc kernel validation succeeded for only 20 of 42 feasible datasets (47.6%), and PhysioNet and TUH families had task macro-F1 of 7.7% and 0.0%.
• Within-subject evaluation consistently outperformed cross-subject evaluation, which is expected but quantified here at 12.6 points.

Limitations

• Quantitative analysis restricted to 53 completed entries; roughly 200 additional drafts are not yet reliable benchmark units.
• NeuroDoc quality is strongly family-dependent; PhysioNet and TUH drafting remains weak.
• Reference PDFs are copyrighted and not redistributed, limiting reproducibility of the drafting pipeline.

Experimental setup

Datasets: BCIC IV-1 BCIC IV-2a BCIC IV-2b LargeMI Meng19 SHU Stroke INRIA P300 HighGamma MigraineDB OpenBMI EEGMAT

Benchmarks: 18 released EEG datasets across 37 tasks (motor imagery, P300, sleep staging, SSVEP/SSAEP, clinical EEG, cognitive)

Metrics: macro-F1 test accuracy above random baseline within-subject vs cross-subject gap

Baselines: MOABB EEGUnity EEGAgent DCA-Bench Pre-Meta Auto-BenchmarkCard

Builds on: BIDS EEG-BIDS Croissant MNE MOABB

Closest papers by meaning

• NeuroAtlas: Benchmarking Foundation Models for Clinical EEG and Brain-Computer Interfaces (NeurIPS 2026 · similarity 0.91)
• NeuroLM: A Universal Multi-task Foundation Model for Bridging the Gap between Language and EEG Signals (ICLR 2025 · similarity 0.88)
• What do EEG Foundation Models Capture from Human Brain Signals? (NeurIPS 2026 · similarity 0.88)
• BrainOmni: A Brain Foundation Model for Unified EEG and MEG Signals (NeurIPS 2025 · similarity 0.88)
• LEAF: Language-EEG Aligned Foundation Model for Brain-Computer Interfaces (NeurIPS 2026 · similarity 0.86)
• NeurIPT: Foundation Model for Neural Interfaces (NeurIPS 2025 · similarity 0.86)