Research map / Vision-language and multimodal / Human and embodied modeling
Neuro-Vision to Language: Enhancing Brain Recording-based Visual Reconstruction and Language Interaction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a ViT3D-based fMRI feature extractor aligned to CLIP and VAE embeddings that feeds an LLM (Llama-3-8B) for brain captioning, reasoning, visual reconstruction via UnCLIP, and GradCAM concept localization on NSD.
Problem
Non-invasive fMRI decoding suffers from inter-subject variability, need for subject-specific models and many trials, and lack of semantic precision and interpretability in visual reconstruction and language interaction.
Approach
A single 3D-patch Transformer encoder over preprocessed fMRI, with two alignment heads (CLIP and VAE), whose penultimate hidden states serve as multimodal tokens for an LLM, enabling captioning, reasoning, reconstruction via UnCLIP and concept localization via GradCAM.
Key results
- On visual reconstruction, the method reaches 96.8% Inception and 97.5% CLIP two-way identification, the best among compared methods, using a single model across subjects.
- On brain captioning, the method achieves BLEU4 15.85 and CIDEr 49.51 versus UMBRAE's 17.17 and 53.87 on the original NSD captions; with BLIP2-generated ground truth, BLEU4 reaches 32.17 and CIDEr 83.94.
- Replacing ViT3D with flattened-patch input drops complex reasoning CIDEr from 217.83 to 150.67 and brain caption BLEU4 from 15.85 to 8.49.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ViT3D rather than flattening and patching fMRI data (ablation) | Our Method · NSD brain caption | BLEU4: 8.49 -> 15.85 | Helped |
| Use ViT3D rather than flattening and patching fMRI data (ablation) | Our Method* · NSD brain caption, BLIP2 captions as ground truth | BLEU4: 21.39 -> 32.17 | Helped |
| Use ViT3D rather than flattening and patching fMRI data (ablation) | Our Method · NSD detail description | BLEU4: 6.13 -> 12.41 | Helped |
| Use ViT3D rather than flattening and patching fMRI data (ablation) | Our Method · NSD complex reasoning | BLEU4: 30.57 -> 36.46 | Helped |
| Remove LLM integration (ablation) | Our Method · NSD visual reconstruction | PixCorr: .265 -> .263 | No clear effect |
| Remove LLM integration (ablation) | Our Method · NSD visual reconstruction | SSIM: .369 -> .357 | Hurt |
| Remove LLM integration (ablation) | Our Method · NSD visual reconstruction | CLIP: 96.1% -> 97.5% | Hurt |
| Remove VAE feature alignment (ablation) | Our Method · NSD visual reconstruction | PixCorr: .093 -> .265 | Helped |
| Remove VAE feature alignment (ablation) | Our Method · NSD visual reconstruction | CLIP: 95.7% -> 97.5% | Helped |
| Remove C_Subj (ablation) | Our Method · NSD visual reconstruction | PixCorr: .241 -> .265 | Helped |
| Remove C_Subj and ViT3D (ablation) | Our Method · NSD visual reconstruction | PixCorr: .164 -> .265 | Helped |
| Use Llama-3-8B instead of Vicuna-13B | Visual reconstruction model · NSD visual reconstruction | CLIP: 96.7% -> 97.0% (brief descriptions); 97.1% -> 97.5% (instruction reconstruction) | Mixed |
Surprising findings
- Removing the LLM component leaves low-level metrics nearly unchanged while high-level gains come mainly from LLM integration, as the paper notes for the beta ablation.
- Single-trial reconstruction shows only a slight drop relative to the trial-averaged setting.
Limitations
- Validated only on NSD with four subjects, limiting generalization to broader populations.
- Substantial computational requirements for LLM integration and ViT3D.
- Single-trial variability from head motion and physiological noise may reduce decoding accuracy.
Experimental setup
Datasets: Natural Scenes Dataset (NSD) COCO
Benchmarks: NSD visual reconstruction (PixCorr, SSIM, AlexNet, Inception, CLIP, EffNet-B, SwAV) brain captioning (BLEU, METEOR, ROUGE, CIDEr, SPICE, CLIP-S) detailed description and complex reasoning on NSD-derived language data
Metrics: PixCorr SSIM AlexNet(2) AlexNet(5) Inception CLIP EffNet-B SwAV BLEU METEOR
Baselines: UMBRAE BrainCap OneLLM SDRecon MindEye DREAM MindBridge Brain-Diffuser UniBrain Mind-Reader
Builds on: LLaVA UnCLIP GradCAM Vision Transformer 3D
Closest papers by meaning
- MoRE-Brain: Routed Mixture of Experts for Interpretable and Generalizable Cross-Subject fMRI Visual Decoding (NeurIPS 2025 · similarity 0.90)
- Multigranular Evaluation for Brain Visual Decoding (AAAI 2026 · similarity 0.90)
- See Through Their Minds: Learning Transferable Brain Decoding Models from Cross-Subject fMRI (AAAI 2025 · similarity 0.90)
- MindAlign: Bridging EEG, Vision, and Language for Zero-Shot Visual Decoding (NeurIPS 2026 · similarity 0.89)
- Brain decoding: toward real-time reconstruction of visual perception (ICLR 2024 · similarity 0.89)
- $i$MIND: Insightful Multi-subject Invariant Neural Decoding (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.