Research map / Label-efficient and robust learning / Recognition and long-tail

SENSE: Semantic Neural Speech Synthesis from Brain Dynamics via Spatial Graph Encoding

NeurIPS 2026 · Label-efficient and robust learning › Recognition and long-tail · approach: Multilingual language representation pretraining · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes SENSE, a graph-based EEG encoder over electrode geometry plus congruency-restricted CLIP semantic conditioning (ESC) for EEG-to-speech waveform synthesis on the N400 dataset.

Problem

EEG-to-speech generation has been framed as acoustic reconstruction only, ignoring semantic content and spatial electrode structure, and existing models treat congruent and incongruent N400 trials identically.

Approach

Combine a graph convolution over BioSemi electrode geometry (with channel gate and CNN skip) with an EEG-to-CLIP alignment loss applied only to congruent trials, and use a conditioning bridge so the EEG embedding can directly condition a VITS decoder at inference.

Key results

• On the unseen-subject split, SENSE reaches WER 0.995 versus 1.123 for FE-Phoneme and MCD 9.93 versus 10.65.
• SENSE trained on only two subjects surpasses FE-Phoneme's 18-subject WER, per the scaling experiment.
• Ablation: removing ESC raises WER from 0.995 to 1.032 and lowers BERTScore from 0.109 to 0.076.

Surprising findings

• Shuffled CLIP targets during ESC training give performance comparable to or slightly below the no-ESC baseline, indicating gains depend on correct semantic targets.
• CLAP, an audio-language encoder, performs worse than text-only SBERT as an ESC target.

Limitations

• WER values near 1 indicate word-level intelligibility remains poor.
• Evaluated only on the controlled N400 corpus; naturalistic transfer is preliminary.
• Channel attribution is model-internal gradient saliency, not neural source localization.

Experimental setup

Datasets: N400 EEG corpus

Benchmarks: N400 EEG corpus

Metrics: MCD Mel-Correlation STOI WER BERTScore CLIP-Sim

Baselines: FESDE FE-Phoneme

Builds on: VITS FESDE FE-Phoneme S4 CLIP

Closest papers by meaning

• MindAlign: Bridging EEG, Vision, and Language for Zero-Shot Visual Decoding (NeurIPS 2026 · similarity 0.87)
• LEAF: Language-EEG Aligned Foundation Model for Brain-Computer Interfaces (NeurIPS 2026 · similarity 0.87)
• Improving Semantic Understanding in Speech Language Models via Brain-tuning (ICLR 2025 · similarity 0.87)
• Brain-OF: An Omnifunctional Foundation Model for fMRI, EEG and MEG (NeurIPS 2026 · similarity 0.86)
• BrainOmni: A Brain Foundation Model for Unified EEG and MEG Signals (NeurIPS 2025 · similarity 0.86)
• EEG Benchmarking Needs a Task Specification Layer: NeuroDoc for Rulebook-Guided, Executable Benchmark Construction (NeurIPS 2026 · similarity 0.86)