Research map / Architectures and efficiency / RNNs and normalization
Time-Masked Transformers with Lightweight Test-Time Adaptation for Neural Speech Decoding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Replaces the baseline GRU in the Brain-to-Text '24 benchmark with a compact causal Transformer trained with heavy time-masking, and adds DietCORP, a one-gradient-step test-time adaptation using time-masked augmentations of each trial.
Problem
Accurate speech neuroprostheses decoding from intracortical recordings has relied on non-streaming, computationally expensive models and LLM rescoring, and performance degrades across held-out days; the paper targets accuracy, streaming, low compute, and robustness together.
Approach
Heavy contiguous time-masking (about 53% of each trial masked on average) as a regularizer, a compact unidirectional Transformer with non-overlapping patches replacing the GRU, and DietCORP, which adapts only the patch embedding with one gradient step over Z time-masked augmentations of the current trial using LM-refined pseudo-labels.
Key results
- The time-masked causal Transformer reaches 12.17% WER with a 3-gram LM versus 15.25% for the baseline GRU (about 20% relative gain), and 8.18% vs 11.12% with the 5-gram setup.
- The Transformer uses 83% fewer parameters (9.4M vs 56.7M), 52% less peak memory than the unidirectional GRU, and 58% shorter per-epoch training time.
- DietCORP keeps WER from rising across held-out days: over five days WER goes from 21.24% to 22.97% versus 22.74% to 32.58% without adaptation, using 18 ms per trial and 1.3 GiB peak memory.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace the baseline unidirectional GRU with a time-masked causal Transformer | Brain-to-Text '24 speech decoding · Test, 3-gram LM | WER (%): 15.25 ± 0.16 -> 12.17 ± 0.22 | Helped |
| Replace the baseline unidirectional GRU with a time-masked causal Transformer | Brain-to-Text '24 speech decoding · Test, 5-gram LM | WER (%): 11.12 ± 0.13 -> 8.18 ± 0.22 | Helped |
| Use the time-masked Transformer instead of the Linderman Lab bidirectional GRU | Brain-to-Text '24 speech decoding · Test, 5-gram LM | WER (%): 8.0 vs 8.18 ± 0.22 | No clear effect |
| Use the time-masked Transformer instead of diphone decoding with a bidirectional GRU | Brain-to-Text '24 speech decoding · Test, 5-gram LM | WER (%): 8.18 ± 0.22 vs 8.39 ± 0.22 | No clear effect |
| Fine-tune Llama 3.1 8B to correct outputs from 10 Transformer seeds | Time-Masked Causal Transformer · Brain-to-Text '24 test, fine-tuned LLM | WER (%): 5.68 vs 5.77 | Helped |
| Adapt on each trial using multiple time-masked augmentations | Time-Masked Causal Transformer · Held-out days, 5 days, 3-gram LM | WER (%): 22.74 ± 0.30 to 32.58 ± 0.60 without adaptation; 21.24 ± 0.29 to 22.97 ± 0.72 with DietCORP | Helped |
| Adapt on each trial using multiple time-masked augmentations | Time-Masked Causal Transformer · Held-out days, 8 days, 3-gram LM | WER (%): 28.87 ± 0.54 to 66.47 ± 0.67 without adaptation; 26.32 ± 0.32 to 31.74 ± 1.41 with DietCORP | Helped |
| Remove neural-data log-transformation (ablation) | Time-Masked Transformer · Validation, 3-gram LM | WER (%): 17.15 ± 0.15 -> 17.85 ± 0.15 | Hurt |
| Remove learning-rate scheduler (ablation) | Time-Masked Transformer · Validation, 3-gram LM | WER (%): 17.15 ± 0.15 -> 17.85 ± 0.21 | Hurt |
| Replace T5-style relative positional embeddings with absolute sinusoidal embeddings (ablation) | Time-Masked Transformer · Validation, 3-gram LM | WER (%): 17.15 ± 0.15 -> 18.89 ± 0.06 | Hurt |
| Remove time-masking (ablation) | Time-Masked Transformer · Validation, 3-gram LM | WER (%): 17.15 ± 0.15 -> 20.17 ± 0.2 | Hurt |
| Replace the Transformer architecture with a GRU while retaining time-masking (ablation) | Time-Masked Transformer · Validation, 3-gram LM | WER (%): 17.15 ± 0.15 -> 17.85 ± 0.12 | Hurt |
Surprising findings
- The time-masked GRU is only about 4% worse than the time-masked Transformer, so most of the gain comes from time-masking rather than architecture alone.
- Non-overlapping inputs substantially hurt the time-masked GRU (validation WER 17.85% to 19.66%), so the GRU still needs overlapping windows.
Limitations
- Results come from a single participant on one open-source MEA dataset.
- The 3-gram LM needs about 60 GB of CPU memory, which limits on-device use.
- Beam search allows revision of previously decoded text, which complicates streaming integration with text-to-speech.
Experimental setup
Datasets: Brain-to-Text '24 benchmark (MEA recordings, ALS participant) Switchboard corpus (text sentences)
Benchmarks: Brain-to-Text '24 benchmark
Metrics: word error rate phoneme error rate peak GPU memory adaptation time parameter count MFLOPS
Baselines: Baseline unidirectional GRU Linderman Lab bidirectional GRU Diphone decoding with bidirectional GRU Baseline GRU with DietCORP
Builds on: CORP Brain-to-Text Benchmark '24 baseline GRU T5 relative positional embeddings SpecAugment-style masking
Closest papers by meaning
- The Brain's Bitter Lesson: Scaling Speech Decoding With Self-Supervised Learning (ICML 2025 · similarity 0.89)
- Du-IN: Discrete units-guided mask modeling for decoding speech from Intracranial Neural signals (NeurIPS 2024 · similarity 0.88)
- Generalizable, real-time neural decoding with hybrid state-space models (NeurIPS 2025 · similarity 0.87)
- Few-shot Algorithms for Consistent Neural Decoding (FALCON) Benchmark (NeurIPS 2024 · similarity 0.87)
- A Scalable, Causal, and Energy Efficient Framework for Neural Decoding with Spiking Neural Networks (NeurIPS 2025 · similarity 0.86)
- LibriBrain100: One Hundred Hours of Broad and Deep MEG Data for Neural Speech Decoding at Scale (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.