Research map / Large language models / Instruction and fine-tuning
Brain-tuning Improves Generalizability and Efficiency of Brain Alignment in Speech Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Fine-tunes pretrained speech models (Wav2Vec2.0, HuBERT) with LoRA to jointly predict fMRI responses from multiple participants ("multi-brain-tuning"), improving brain alignment efficiency and generalization to new participants and datasets.
Problem
Brain alignment estimation and improvement for pretrained speech/language models are participant-dependent and data-hungry, which limits generalization to new participants and population-level analyses.
Approach
Instead of per-participant brain-tuning, jointly fine-tune a speech model with LoRA and a shared FreeSurfer-ROI projection head on fMRI responses from multiple participants aligned to a common cortical space, binding participants by shared stimuli and computing the L2 loss per participant sequentially.
Key results
- Multi-brain-tuned models reach max pretrained brain alignment with about one-fifth of encoding data on held-out participants, versus about twice that for single-brain-tuned models.
- Brain alignment improves by up to 50% over max pretrained alignment on training participants with full encoding data.
- Downstream phoneme and phonetic sentence type F1 never falls below pretrained, and multi-brain-tuning eventually matches the LLM-tuned baseline.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Jointly fine-tune on multiple participants' fMRI responses rather than one participant | Speech models · Moth Radio Hour held-out participants | brain alignment | Helped |
| Use Multi-brain-tuning rather than the pretrained model | Wav2Vec2.0 and HuBERT · Moth Radio Hour training participants | brain alignment: up to 50% | Helped |
| Use Multi-brain-tuning rather than the pretrained model | Wav2Vec2.0 and HuBERT · Moth Radio Hour held-out participants | encoding data required to reach pretrained alignment: approximately one-fifth of the data | Helped |
| Use Multi-brain-tuning rather than Single-brain-tuning | Speech models · Moth Radio Hour held-out participants | encoding data required to reach pretrained alignment: approximately twice this amount | Helped |
| Tune on multiple Moth Radio Hour participants instead of the pretrained model | Wav2Vec2.0 · Narratives | brain alignment | Helped |
| Increase the amount of brain-tuning data | Multi-brain-tuning · Moth Radio Hour held-out participants | brain alignment | Helped |
| Increase the amount of brain-tuning data | Multi-brain-tuning · TIMIT | F1-score | Helped |
| Use Multi-brain-tuning rather than stimulus-only tuning | Wav2Vec2.0 · Moth Radio Hour training participants | brain alignment | Helped |
| Use Multi-brain-tuning rather than LLM-tuning | Wav2Vec2.0 · Moth Radio Hour training participants | brain alignment | Helped |
| Use LoRA rank 8 rather than a higher rank or full-model tuning (ablation) | Multi-brain-tuning · Moth Radio Hour | brain alignment | Helped |
| Use L2 reconstruction loss rather than correlation or cosine plus L2 loss (ablation) | Multi-brain-tuning · Moth Radio Hour | brain alignment | Mixed |
| Compute and backpropagate each participant's loss independently rather than averaging losses (ablation) | Multi-brain-tuning · Moth Radio Hour | brain alignment | Helped |
Surprising findings
- Auditory cortex alignment slightly decreases after brain-tuning on upper-middle and later layers, attributed to a semantic shift.
- The L2 loss scales better with data, while the correlation loss performs better at low tuning data sizes (<=6h).
Limitations
- Focused only on language-related brain regions and auditory ROIs.
- Experiments are limited to English and to large public fMRI datasets.
- Loss function choice strongly affects results and no loss is clearly best across data sizes.
Experimental setup
Datasets: Moth Radio Hour Narratives TIMIT
Benchmarks: Moth Radio Hour Narratives TIMIT
Metrics: normalized Pearson correlation brain alignment F1-score
Baselines: Single-brain-tuned LLM-tuned Stimulus-tuned Narratives-Multi-brain-tuned Shared Response Modeling
Builds on: brain-tuning BrainWavLM LoRA HuBERT Wav2Vec 2.0
Closest papers by meaning
- Improving Semantic Understanding in Speech Language Models via Brain-tuning (ICLR 2025 · similarity 0.94)
- Scaling and context steer LLMs along the same computational path as the human brain (NeurIPS 2025 · similarity 0.89)
- fMRI predictors based on language models of increasing complexity recover brain left lateralization (NeurIPS 2024 · similarity 0.88)
- Correlating instruction-tuning (in multimodal models) with vision-language processing (in the brain) (ICLR 2025 · similarity 0.88)
- Hierarchical Frequency Tagging Probe (HFTP): A Unified Approach to Investigate Syntactic Structure Representations in Large Language Models and the Human Brain (NeurIPS 2025 · similarity 0.88)
- Far from the Shallow: Brain-Predictive Reasoning Embedding through Residual Disentanglement (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.