AI Paper Atlas

Research map / Vision-language and multimodal / Cross-modal learning

Do Speech BCIs Need Larger Models? Rethinking Neural Decoding beyond Scaling

Zehui Feng, Weichuan Wang, Xiaohan Chen, Cuntai Guan, Ting Han

NeurIPS 2026 · Vision-language and multimodal › Cross-modal learning · approach: Multimodal fusion with missing modalities · 0 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Questions whether large models are needed for speech BCI neural decoding and proposes a compact model (BEST) as an alternative to scaled, multi-stage pipelines.

Problem

Speech BCI neural decoding increasingly uses large models and complex multi-stage pipelines, but training data are scarce, noisy, and expensive, so the benefit of model scaling is unclear.

Approach

Argues that compact, purpose-built decoding models can match large models in speech BCI, questioning the scaling assumption; the proposed model is named BEST (full method details unavailable in the provided abstract).

Measured effects reported in the paper

What was changedWhereMeasuredEffect
replace phoneme-only pipeline with hierarchical phoneme-character fusion in lightweight encoder (ablation)BEST · Brain-to-Text '24 T12PER: 0.1628 -> 0.1241Improved
add phoneme-character fusion module (ablation)BEST · Brain-to-Text '24 T12PER: 0.1321 -> 0.1241Improved
use two-stage training instead of end-to-end (ablation)BEST · Brain-to-Text '25 T15 validationWER: 0.2071 -> 0.0681Improved
add multi-task loss to end-to-end training (ablation)BEST end-to-end · Brain-to-Text '24 T12WER: 0.3415 -> 0.2253Improved
prevent over-fitting (well-fitted instead of over-fitted) (ablation)BEST encoder · Brain-to-Text '25 T15 validationWER: 0.0803 -> 0.0681Improved
add masked autoencoder pretraining (ablation)BEST · Brain-to-Text '24 T12PER: 0.1321 -> 0.1420Hurt
scale decoder from Whisper-small (244M) to larger text LLMs such as Llama3.1 8B (ablation)BEST decoder · Brain-to-Text '25 T15 validationValid WER: 0.0681 -> 0.1131Hurt
use full fine-tuning instead of partial fine-tuning of decoder (ablation)Whisper-small decoder · Brain-to-Text '25 T15 validationValid WER: 0.0681 -> 0.0764Hurt
use cross-attention for phoneme-character fusion instead of multi-scale convolution (ablation)BEST fusion · Brain-to-Text '25 T15WER: 0.0681 -> 0.4219Hurt
increase encoder dimension from 768 to 1024 (ablation)BEST encoder · Brain-to-Text '24 T12PER: 0.1321 -> 0.2063Hurt
add 5-gram LM rescoring to two-stage BEST (ablation)BEST · Brain-to-Text '25 T15 validationWER: 0.0681 -> 0.0257Improved
replace BEST encoder with Transformer encoder under same 5-gram LM (ablation)BEST · Brain-to-Text '25 T15 validationWER: 0.0257 -> 0.0381Hurt

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy