Research map / Large language models / Language, safety and interpretability
Decoding Inner Speech with an End-to-End Brain-to-Text Neural Interface
Yizi Zhang, Linyang He, Chaofei Fan, Tingkai Liu, Han Yu, Trung Le, Jingyuan Li, Scott Linderman, Lea Duncker, Francis R Willett, Nima Mesgarani, Liam Paninski
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace the RNN encoder with a pretrained transformer encoder in the cascaded decoder | BIT Cascaded · Brain-to-Text '24 holdout | WER: 9.76% -> 6.35% | Improved |
| add LLM-merger ensemble to the cascaded decoder | BIT Cascaded · Brain-to-Text '24 holdout | WER: 6.35% -> 5.10% | Improved |
| replace RNN encoder plus text LLM with pretrained transformer plus audio-LLM end-to-end | BIT End-to-End + Ensemble · Brain-to-Text '24 holdout | WER: 24.69% -> 10.22% | Improved |
| add LLM-merger ensemble to the end-to-end decoder | BIT End-to-End · Brain-to-Text '25 holdout | WER: 11.06% -> 7.76% | Improved |
| add LLM-merger ensemble to the cascaded decoder | BIT Cascaded · Brain-to-Text '25 holdout | WER: 4.06% -> 1.76% | Improved |
| add SBP features to thresholded spikes (ablation) | BIT encoder · Attempted speech T12 validation | PER: 21.35% -> 17.26% | Improved |
| add SSL pretraining (BIT-Human, BIT-All) over training the transformer from scratch (ablation) | BIT-TFS · Imagined speech (50-word vocabulary) | WER: 39-45% | Improved |
| add SSL pretraining over transformer from scratch (ablation) | BIT-TFS · Attempted speech T12 validation | PER: 17.26% -> 14.39% | Improved |
| add monkey data to human-only pretraining (ablation) | BIT-Human · Imagined speech T12 validation | PER: 19.63% -> 18.08% | Improved |
| use cross-subject label-free SSL pretraining instead of within-subject cross-task supervised pretraining (ablation) | BIT-Cross-Task-Only · Imagined speech T12 validation | PER: 20.46% -> 18.08% | Improved |
| use audio-LLM instead of text LLM of comparable size (ablation) | BIT End-to-End · Attempted speech validation | WER | Improved |
| treat neural outputs as neural modality instead of audio (ablation) | audio-LLM decoder · Attempted speech validation | WER | Improved |
Closest papers by meaning
- The Brain's Bitter Lesson: Scaling Speech Decoding With Self-Supervised Learning (ICML 2025 · similarity 0.90)
- Du-IN: Discrete units-guided mask modeling for decoding speech from Intracranial Neural signals (NeurIPS 2024 · similarity 0.89)
- LibriBrain100: One Hundred Hours of Broad and Deep MEG Data for Neural Speech Decoding at Scale (NeurIPS 2026 · similarity 0.88)
- Time-Masked Transformers with Lightweight Test-Time Adaptation for Neural Speech Decoding (NeurIPS 2025 · similarity 0.88)
- Improving Semantic Understanding in Speech Language Models via Brain-tuning (ICLR 2025 · similarity 0.88)
- Brain-tuning Improves Generalizability and Efficiency of Brain Alignment in Speech Models (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
