Research map / Large language models / Language, safety and interpretability
Improving Semantic Understanding in Speech Language Models via Brain-tuning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Fine-tunes Wav2vec2.0, HuBERT, and the Whisper encoder to predict fMRI responses to natural stories ("brain-tuning"), showing improved late-language-region brain alignment, reduced low-level feature reliance, and better downstream semantic task performance.
Problem
Speech language models align with brain responses mostly through low-level speech features rather than semantics, limiting their use as model organisms of semantic processing and their semantic understanding on downstream tasks.
Approach
Fine-tune pretrained speech models with a linear projection head to regress fMRI responses to the audio stimuli (brain-tuning), with controls using block-permuted fMRI and a larger speech model's representations as targets, then evaluate alignment, low-level feature impact, and downstream semantic tasks.
Key results
- Brain-tuning raised late language region alignment by about 30% over pretrained Wav2vec2.0 and HuBERT, while Whisper showed no significant gain there.
- Brain-tuning reduced low-level feature impact in late language regions across all three model families.
- Brain-tuned Whisper encoder improved word identity prediction by 17% and phonetic sentence-type prediction by 20-23%, with ASR accuracy up to 12% higher.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune Wav2vec2.0 with matched fMRI responses | Wav2vec2.0 · Late language regions | normalized brain alignment: 30% | Helped |
| Fine-tune HuBERT with matched fMRI responses | HuBERT · Late language regions | normalized brain alignment: 30% | Helped |
| Fine-tune Whisper encoder with matched fMRI responses | Whisper encoder · Late language regions | normalized brain alignment | No clear effect |
| Fine-tune speech models with matched fMRI responses | Speech models · Primary auditory regions | normalized brain alignment | No clear effect |
| Fine-tune speech models with matched fMRI responses | Speech models · Late language regions | low-level feature impact | Helped |
| Fine-tune speech models with matched fMRI responses | Speech models · Primary auditory regions | low-level feature impact | No clear effect |
| Fine-tune with matched fMRI responses | Speech models · English ASR (TIMIT) | accuracy (1-WER): up to 12% | Helped |
| Fine-tune with matched fMRI responses | Speech models · Phonetic sentence-type prediction (TIMIT) | F1-score: 20-23% | Helped |
| Fine-tune with matched fMRI responses | Wav2vec2.0 and HuBERT · Sequence understanding (SLURP) | F1-score | Helped |
| Fine-tune with matched fMRI responses | Speech models · Phonemes prediction (TIMIT) | F1-score: 6% | Helped |
| Fine-tune with matched fMRI responses | Wav2vec2.0 and HuBERT · Word identity prediction (Speech Commands) | F1-score: 3% | Helped |
| Fine-tune with matched fMRI responses | Whisper encoder · Word identity prediction (Speech Commands) | F1-score: 17% | Helped |
Surprising findings
- Brain-tuned Whisper encoder improved substantially on downstream tasks despite no significant gain in brain alignment in late language regions.
- Random-fMRI (block-permuted) brain-tuning lowered alignment below the pretrained models, showing correct fMRI targets matter.
Limitations
- Brain-tuning only uses fMRI from 8 participants listening to 27 stories, so the training signal is limited.
- Whisper showed no significant brain alignment gain in late language regions, so the effect is not universal across model families.
- Fine-tuning is done separately per participant, so the resulting models are participant-specific.
Experimental setup
Datasets: Moth Radio Hour fMRI dataset TIMIT CREMA-D Speech Commands SLURP Glasser atlas
Benchmarks: TIMIT CREMA-D Speech Commands SLURP
Metrics: normalized brain alignment low-level impact (percentage drop in alignment) F1-score 1-WER semantic-phonetic preference
Baselines: Pretrained Wav2vec2.0 Pretrained HuBERT Pretrained Whisper encoder BigSLM fine-tuned (Whisper Medium representations) Random-fMRI brain-tuning HuBERT large
Builds on: Wav2vec2.0 HuBERT Whisper BERT brain-tuning (Schwartz et al.)
Closest papers by meaning
- Brain-tuning Improves Generalizability and Efficiency of Brain Alignment in Speech Models (NeurIPS 2025 · similarity 0.94)
- Far from the Shallow: Brain-Predictive Reasoning Embedding through Residual Disentanglement (NeurIPS 2025 · similarity 0.88)
- Divergences between Language Models and Human Brains (NeurIPS 2024 · similarity 0.88)
- Scaling and context steer LLMs along the same computational path as the human brain (NeurIPS 2025 · similarity 0.88)
- Hierarchical Frequency Tagging Probe (HFTP): A Unified Approach to Investigate Syntactic Structure Representations in Large Language Models and the Human Brain (NeurIPS 2025 · similarity 0.87)
- fMRI predictors based on language models of increasing complexity recover brain left lateralization (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.