Research map / Vision-language and multimodal / Multimodal LLMs

Multigranular Evaluation for Brain Visual Decoding

AAAI 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Instruction-tuned multimodal assistants · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes BASIC, a multigranular evaluation framework for brain visual decoding combining MLLM-based semantic graph matching (BASIC-H) and mask-based hierarchical segmentation matching (BASIC-L), and benchmarks many decoders across six stimulus-neuroimaging datasets.

Problem

Existing brain visual decoding metrics saturate across methods, lack neuroscientific grounding, and fail to capture fine-grained structural and semantic differences between decoded and ground-truth images.

Approach

Evaluate decoded images on two axes: high-level semantic similarity via MLLM-generated captions parsed into objects, attributes and relations with precision/recall/F1 matching, and low-level structural similarity via Grounded-SAM2 masks compared at foreground, binary, semantic, instance and part granularities.

Key results

• On NSD, NeuroVLA achieves the top BASIC-H score of 47.88, ahead of DREAM at 46.37.
• On NSD, NeuroPictor leads BASIC-L with 25.88, while STTM scores 22.90 and MindEye2 scores 22.16.
• Prior NSD metrics saturate, with many methods' scores clustered closely together; BASIC is reported to provide clearer differentiation across methods.

Surprising findings

• NeuroVLA scores highest on BASIC-H yet among the lowest on BASIC-L (13.54), showing the two axes disagree.

Limitations

• Relies on MLLM captions and parsing, which can hallucinate; ground-truth captions were human-corrected only for NSD-style evaluation.
• Video and 3D data evaluated via representative frames and rendered views rather than modality-specific tools.

Experimental setup

Datasets: NSD EEG-Things CC2017 SEED-DV fMRI-Shape EEG-3D COCO

Benchmarks: NSD EEG-Things CC2017 SEED-DV fMRI-Shape EEG-3D

Metrics: precision recall F1 IoU average precision BASIC-H BASIC-L

Baselines: PixCorr SSIM AlexNet n-way Inception CLIP EffNet SwAV n-way classification accuracy MMR

Builds on: Grounded-SAM2 LLaVA-1.6 Natural Scenes Dataset

Closest papers by meaning

• Neuro-Vision to Language: Enhancing Brain Recording-based Visual Reconstruction and Language Interaction (NeurIPS 2024 · similarity 0.90)
• $i$MIND: Insightful Multi-subject Invariant Neural Decoding (NeurIPS 2025 · similarity 0.89)
• Breaking the Illusion: When Positive Meets Negative in Multimodal Decoding (CVPR 2026 · similarity 0.89)
• Toward Generalizing Visual Brain Decoding to Unseen Subjects (ICLR 2025 · similarity 0.89)
• MindAlign: Bridging EEG, Vision, and Language for Zero-Shot Visual Decoding (NeurIPS 2026 · similarity 0.88)
• Brain decoding: toward real-time reconstruction of visual perception (ICLR 2024 · similarity 0.88)