Research map / Vision-language and multimodal / Cross-modal learning
Do Speech BCIs Need Larger Models? Rethinking Neural Decoding beyond Scaling
Zehui Feng, Weichuan Wang, Xiaohan Chen, Cuntai Guan, Ting Han
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Questions whether large models are needed for speech BCI neural decoding and proposes a compact model (BEST) as an alternative to scaled, multi-stage pipelines.
Problem
Speech BCI neural decoding increasingly uses large models and complex multi-stage pipelines, but training data are scarce, noisy, and expensive, so the benefit of model scaling is unclear.
Approach
Argues that compact, purpose-built decoding models can match large models in speech BCI, questioning the scaling assumption; the proposed model is named BEST (full method details unavailable in the provided abstract).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace phoneme-only pipeline with hierarchical phoneme-character fusion in lightweight encoder (ablation) | BEST · Brain-to-Text '24 T12 | PER: 0.1628 -> 0.1241 | Improved |
| add phoneme-character fusion module (ablation) | BEST · Brain-to-Text '24 T12 | PER: 0.1321 -> 0.1241 | Improved |
| use two-stage training instead of end-to-end (ablation) | BEST · Brain-to-Text '25 T15 validation | WER: 0.2071 -> 0.0681 | Improved |
| add multi-task loss to end-to-end training (ablation) | BEST end-to-end · Brain-to-Text '24 T12 | WER: 0.3415 -> 0.2253 | Improved |
| prevent over-fitting (well-fitted instead of over-fitted) (ablation) | BEST encoder · Brain-to-Text '25 T15 validation | WER: 0.0803 -> 0.0681 | Improved |
| add masked autoencoder pretraining (ablation) | BEST · Brain-to-Text '24 T12 | PER: 0.1321 -> 0.1420 | Hurt |
| scale decoder from Whisper-small (244M) to larger text LLMs such as Llama3.1 8B (ablation) | BEST decoder · Brain-to-Text '25 T15 validation | Valid WER: 0.0681 -> 0.1131 | Hurt |
| use full fine-tuning instead of partial fine-tuning of decoder (ablation) | Whisper-small decoder · Brain-to-Text '25 T15 validation | Valid WER: 0.0681 -> 0.0764 | Hurt |
| use cross-attention for phoneme-character fusion instead of multi-scale convolution (ablation) | BEST fusion · Brain-to-Text '25 T15 | WER: 0.0681 -> 0.4219 | Hurt |
| increase encoder dimension from 768 to 1024 (ablation) | BEST encoder · Brain-to-Text '24 T12 | PER: 0.1321 -> 0.2063 | Hurt |
| add 5-gram LM rescoring to two-stage BEST (ablation) | BEST · Brain-to-Text '25 T15 validation | WER: 0.0681 -> 0.0257 | Improved |
| replace BEST encoder with Transformer encoder under same 5-gram LM (ablation) | BEST · Brain-to-Text '25 T15 validation | WER: 0.0257 -> 0.0381 | Hurt |
Closest papers by meaning
- The Brain's Bitter Lesson: Scaling Speech Decoding With Self-Supervised Learning (ICML 2025 · similarity 0.90)
- Du-IN: Discrete units-guided mask modeling for decoding speech from Intracranial Neural signals (NeurIPS 2024 · similarity 0.89)
- Brain-tuning Improves Generalizability and Efficiency of Brain Alignment in Speech Models (NeurIPS 2025 · similarity 0.88)
- OWLS: Scaling Laws for Multilingual Speech Recognition and Translation Models (ICML 2025 · similarity 0.87)
- LibriBrain100: One Hundred Hours of Broad and Deep MEG Data for Neural Speech Decoding at Scale (NeurIPS 2026 · similarity 0.87)
- Improving Semantic Understanding in Speech Language Models via Brain-tuning (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
