Research map / Architectures and efficiency / Attention and transformers
Du-IN: Discrete units-guided mask modeling for decoding speech from Intracranial Neural signals
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Collects a 12-subject Chinese word-reading sEEG dataset and proposes Du-IN, a region-level-token Transformer pretrained with VQ-VAE discrete codex-guided masked modeling, reaching 62.70% top-1 on 61-word classification.
Problem
Speech decoding from sEEG is underexplored, and existing intracranial/EEG pre-training methods use channel-level or brain-level tokens and are not evaluated on group-level, language-region-specific tasks like speech decoding.
Approach
Fuse channels within specific brain regions into region-level tokens via 1D depthwise convolution over short (100 ms) patches, then pretrain with a VQ-VAE-derived discrete neural codex as masked-token prediction targets (Du-IN MAE).
Key results
- Du-IN (mae) reaches 62.70% ± 4.69% top-1 accuracy on the 61-word classification task, the best among all compared methods.
- Region-level baselines such as Neuro-BERT (49.51%) and EEG-Conformer (45.82%) outperform channel-level brain foundation models (Brant 12.42%, LaBraM 11.53%).
- Electrode selection (12.25 channels on average) raises accuracy from 30.12% to 55.92% for Du-IN.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Du-IN MAE initialization instead of training from scratch (ablation) | Du-IN · Chinese word-reading sEEG, 61-word classification | accuracy: 56.29$\pm$5.20 -> 62.70$\pm$4.69 | Helped |
| Initialize from VQ-VAE and insert its vector quantizer (ablation) | Du-IN · Chinese word-reading sEEG, 61-word classification | accuracy: 56.29$\pm$5.20 -> 44.17$\pm$4.04 | Hurt |
| Initialize the encoder from VQ-VAE without inserting the vector quantizer (ablation) | Du-IN · Chinese word-reading sEEG, 61-word classification | accuracy: 56.29$\pm$5.20 -> 58.24$\pm$4.83 | Helped |
| Select electrodes from speech-related brain regions instead of using all electrodes (ablation) | Du-IN · Chinese word-reading sEEG, 61-word classification | accuracy: 30.12$\pm$5.64 -> 55.92$\pm$4.96 | Helped |
| Include downstream task recordings in pre-training (ablation) | Du-IN (mae) · Chinese word-reading sEEG, 61-word classification | accuracy: 60.02$\pm$4.34 -> 62.70$\pm$4.69 | Helped |
| Use Du-IN MAE initialization instead of VQ-VAE initialization (ablation) | Du-IN · Chinese word-reading sEEG, 61-word classification | accuracy: 58.24$\pm$4.83 -> 62.70$\pm$4.69 | Helped |
| Vary the discrete codex size (ablation) | Du-IN VQ-VAE and Du-IN MAE · Chinese word-reading sEEG, 61-word classification | accuracy | Mixed |
| Vary the codex dimension (ablation) | Du-IN VQ-VAE and Du-IN MAE · Chinese word-reading sEEG, 61-word classification | accuracy | Helped |
| Vary the receptive field of the spatial encoder (ablation) | Du-IN · Chinese word-reading sEEG, 61-word classification | accuracy | Mixed |
| Use discrete codex-guided mask modeling pre-training rather than no pre-training | Du-IN · Chinese word-reading sEEG, 61-word classification | accuracy: 56.29$\pm$5.20 -> 62.70$\pm$4.69 | Helped |
Surprising findings
- Channel-level brain foundation models (Brant, LaBraM, LaBraM-PopT) perform near chance (about 11-12%) on the speech decoding task, far below region-level methods.
- Using only about one electrode's worth of channels in vSMC and STG achieves the best decoding performance.
Limitations
- Restricted to closed-set decoding of 61 pre-determined words.
- Only vocal production (speech) decoding evaluated; semantic-level decoding not addressed.
- Results from a single dataset of 12 subjects with high variance (standard errors around 4-5%).
Experimental setup
Datasets: Chinese word-reading sEEG dataset (61 words, 12 subjects)
Benchmarks: 61-word classification (sEEG word-reading dataset)
Metrics: top-1 accuracy standard error
Baselines: TS-TCC CNN-BiGRU EEG-Conformer Neuro-BERT DeWave BrainBERT Brant LaBraM LaBraM-PopT
Builds on: VQ-VAE LaBraM BrainBERT Brant masked autoencoder
Closest papers by meaning
- Time-Masked Transformers with Lightweight Test-Time Adaptation for Neural Speech Decoding (NeurIPS 2025 · similarity 0.88)
- Towards Homogeneous Lexical Tone Decoding from Heterogeneous Intracranial Recordings (ICLR 2025 · similarity 0.88)
- Neuro-Vision to Language: Enhancing Brain Recording-based Visual Reconstruction and Language Interaction (NeurIPS 2024 · similarity 0.88)
- The Brain's Bitter Lesson: Scaling Speech Decoding With Self-Supervised Learning (ICML 2025 · similarity 0.87)
- BaRISTA: Brain Scale Informed Spatiotemporal Representation of Human Intracranial Neural Activity (NeurIPS 2025 · similarity 0.87)
- Neural decoding from stereotactic EEG: accounting for electrode variability across subjects (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.