Research map / Vision-language and multimodal / Multimodal LLMs
Correlating instruction-tuning (in multimodal models) with vision-language processing (in the brain)
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Probes instruction-tuned multimodal LLMs (InstructBLIP, mPLUG-Owl, IDEFICS) with 10 natural instructions to test whether their text-output embeddings predict fMRI activity in the NSD dataset better than vision-only and multimodal baselines.
Problem
Unknown whether instruction-tuned multimodal LLMs, when prompted with natural task instructions, yield representations that better align with human visual cortex activity and capture instruction-specific information.
Approach
Extract mean-pooled hidden-state embeddings of MLLM-generated outputs under 10 task instructions and use voxelwise ridge encoding models on NSD fMRI, normalized by cross-subject prediction accuracy, with ROI analysis, instruction-wise comparison, and variance partitioning.
Key results
- MLLMs show significantly better normalized brain alignment than ViT-H and perform on par with or better than CLIP-Text across whole visual cortex.
- MLLM alignment reaches about 0.8 in floc-places regions versus about 0.6 in early visual ROIs.
- Most explained variance is shared across instructions, with image captioning overlapping strongly with other prompts.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use instruction-tuned MLLM representations instead of vision-only ViT-H representations | MLLM brain encoding · NSD visual cortex | normalized brain alignment | Helped |
| Use instruction-tuned MLLM representations instead of CLIP-Text representations | MLLM brain encoding · NSD visual cortex | normalized brain alignment | No clear effect |
| Use pretrained rather than randomly initialized MLLM weights | InstructBLIP, mPLUG-Owl, and IDEFICS · NSD visual cortex | normalized brain alignment | Helped |
| Use instruction-tuned InstructBLIP instead of non-instruction-tuned BLIP-2 | InstructBLIP brain encoding · NSD visual cortex, image captioning and visual relationship tasks | normalized brain alignment | Helped |
| Use image-captioning instruction rather than other prompts | InstructBLIP and IDEFICS · NSD EBA, PPA, and FFA | normalized brain alignment | Helped |
| Use image-understanding instructions rather than other prompts | InstructBLIP and mPLUG-Owl · NSD early visual regions | normalized brain alignment | Helped |
| Use task-specific instructions across visual concepts | MLLMs · NSD visual cortex, count and recognition concepts | normalized brain alignment | Helped |
| Use different task instructions for color, positional, and scene concepts | MLLMs · NSD visual cortex | normalized brain alignment | No clear effect |
| Use middle-layer rather than later-layer representations | InstructBLIP and IDEFICS · NSD high-level visual regions | normalized brain alignment | Helped |
| Use later-layer rather than earlier-layer representations | mPLUG-Owl · NSD high-level and early visual regions | normalized brain alignment | Helped |
| Use captioning and other prompts together | InstructBLIP · NSD visual cortex | shared explained variance | Helped |
| Use image-understanding IU2 or scene-recognition SR rather than broader prompts | InstructBLIP · NSD visual cortex | shared explained variance | Hurt |
Surprising findings
- Non-instruction-tuned BLIP-2 aligns only marginally below CLIP-Text, and text-only LLaMA-2-7B aligns closer to vision-only ViT-H.
- Not all instructions improve brain alignment; image captioning and VQ2 are more informative than others.
Limitations
- Cross-subject noise ceiling assumes a perfect model, which may not match real conditions.
- Analysis restricted to four NSD participants and three MLLMs, with pairwise-only variance partitioning.
Experimental setup
Datasets: Natural Scenes Dataset (NSD) COCO
Benchmarks: Natural Scenes Dataset (NSD)
Metrics: Pearson correlation normalized brain alignment shared/unique variance
Baselines: ViT-H CLIP-Text randomly initialized MLLMs BLIP-2 LLaMA-2-7B
Builds on: MultiInstruct
Closest papers by meaning
- Seeing is Understanding: Unlocking Causal Attention into Modality-Mutual Attention for Multimodal LLMs (ICML 2026 · similarity 0.89)
- Revealing Vision-Language Integration in the Brain with Multimodal Networks (ICML 2024 · similarity 0.88)
- Implicit Multimodal Alignment: On the Generalization of Frozen LLMs to Multimodal Inputs (NeurIPS 2024 · similarity 0.88)
- Analyzing Fine-Grained Alignment and Enhancing Vision Understanding in Multimodal Language Models (NeurIPS 2025 · similarity 0.88)
- Learning to Instruct for Visual Instruction Tuning (NeurIPS 2025 · similarity 0.88)
- The Alignment Illusion in Multimodal Large Language Models (NeurIPS 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.