Research map / Vision-language and multimodal / Human and embodied modeling
MindLLM: A Subject-Agnostic and Versatile Model for fMRI-to-text Decoding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MindLLM, a subject-agnostic fMRI encoder with neuroscience-informed attention (learnable queries, positional and parcellation key embeddings, voxel values) feeding a frozen Vicuna-7B, trained with Brain Instruction Tuning on a 980K-conversation MSCOCO-linked dataset.
Problem
Decoding fMRI signals into text suffers from poor predictive performance, narrow task variety, and poor generalization across subjects with varying voxel counts.
Approach
A cross-attention fMRI encoder whose keys come only from voxel coordinates (Fourier positional encoding) plus brain parcellation region embeddings, with voxel fMRI values as the values, so every voxel is used and shared functional priors transfer across subjects; combined with instruction tuning over diverse VQA/caption/memory tasks linked via NSD stimulus images.
Key results
- Versatile decoding on subject 1 yields an average 12.0% improvement over baselines per the paper, and instruction tuning gives a 28.0% average gain over from-scratch models.
- Unseen subject generalization (train on subjects 1-7, test on subject 8) reaches 24.5% average improvement over the second-best subject-agnostic model.
- Brain captioning on subject 1 gives BLEU-4 21.24 and CIDEr 60.97 versus UMBRAE's CIDEr 61.06, with UMBRAE slightly higher on CIDEr and SPICE.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use MindLLM instead of the strongest reported baseline | fMRI-to-text decoding · Brain captioning | BLEU-1: 59.44 -> 61.75 | Helped |
| use MindLLM instead of the strongest reported baseline | subject-agnostic decoding · COCO-QA | Accuracy: 45.33% -> 48.19% | Helped |
| use MindLLM instead of the strongest reported baseline | subject-agnostic decoding · VG-QA | Accuracy: 24.02% -> 24.06% | No clear effect |
| use MindLLM instead of the strongest reported baseline | subject-agnostic decoding · VQA-v2 | Accuracy: 51.23% -> 52.14% | Helped |
| use MindLLM instead of the strongest reported baseline | subject-agnostic decoding · A-OKVQA | Accuracy: 50.44% -> 52.21% | Helped |
| use MindLLM instead of the strongest reported baseline | subject-agnostic decoding · ST-VQA | ANLS: 12.76% -> 12.92% | Helped |
| use MindLLM instead of the strongest reported baseline | subject-agnostic decoding · OK-VQA | Accuracy: 32.30% -> 33.33% | Helped |
| use MindLLM instead of MindBridge | subject-agnostic decoding · COCO-QA, held-out subject 8 | Accuracy: 35.88 -> 38.96 | Helped |
| use MindLLM instead of UniBrain | TDIUC task adaptation · TDIUC | H-MPT: 36.76% -> 45.43% | Helped |
| add Brain Instruction Tuning (ablation) | MindLLM · TDIUC task adaptation | H-MPT: 19.38% -> 45.43% | Helped |
| add region encodings to positional encoding (ablation) | neuroscience-informed attention · COCO-Caption | BLEU-1: 57.03 -> 58.98 | Helped |
| remove fMRI values from the key embeddings (ablation) | neuroscience-informed attention · COCO-Caption | BLEU-1: 54.86 -> 57.03 | Helped |
Surprising findings
- UMBRAE scores higher than MindLLM on CIDEr (61.06 vs 60.97) and SPICE in the brain captioning table.
- Replacing positional encoding with region encodings alone converges faster early but is eventually outperformed by positional encoding.
Limitations
- Focuses on static fMRI without temporal dynamics.
- Results are reported mainly for subject 1 and a held-out subject; cross-subject generalization is still below potential, with domain adaptation left to future work.
Experimental setup
Datasets: Natural Scenes Dataset COCO Caption Image Paragraph Captioning COCO-QA Visual Genome QA VQAv2 OK-VQA ST-VQA TallyQA VQA-E A-OKVQA FSVQA
Benchmarks: COCO Caption COCO-QA VG-QA VQA-v2 A-OKVQA ST-VQA OK-VQA TallyQA VQA-E FSVQA TDIUC
Metrics: BLEU METEOR ROUGE CIDEr SPICE VQA accuracy ANLS RMSE
Baselines: SDRecon OneLLM UniBrain BrainCap BrainChat UMBRAE MindBridge
Builds on: UMBRAE MindBridge UniBrain LLaVA
Closest papers by meaning
- MindBridge: A Cross-Subject Brain Decoding Framework (CVPR 2024 · similarity 0.91)
- MindAligner: Explicit Brain Functional Alignment for Cross-Subject Visual Decoding from Limited fMRI Data (ICML 2025 · similarity 0.89)
- See Through Their Minds: Learning Transferable Brain Decoding Models from Cross-Subject fMRI (AAAI 2025 · similarity 0.88)
- $i$MIND: Insightful Multi-subject Invariant Neural Decoding (NeurIPS 2025 · similarity 0.88)
- Neuro-Vision to Language: Enhancing Brain Recording-based Visual Reconstruction and Language Interaction (NeurIPS 2024 · similarity 0.87)
- MindTuner: Cross-Subject Visual Decoding with Visual Fingerprint and Semantic Correction (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.