Research map / Vision-language and multimodal / CLIP and zero-shot

What Does the Brain See? Multiview Neural Representation to Demystify the Brain-Visual Alignment

NeurIPS 2026 · Vision-language and multimodal › CLIP and zero-shot · approach: Attribute-based zero-shot learning

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes a multiview EEG encoder (state-space temporal, learnable wavelet spectral, attention-modulated GCN spatial) aligned with frozen CLIP-ViT-H-14 image embeddings via contrastive learning with two EEG-specific regularizers, reporting state-of-the-art 200-way zero-shot decoding on THINGS-EEG.

Problem

Zero-shot visual decoding from EEG is hampered by low SNR, non-stationarity and coarse spatial resolution, and existing EEG-vision alignment methods use holistic embeddings that obscure temporal, spectral and spatial structure.

Approach

Instead of a holistic EEG embedding, encode EEG along three complementary views (input-conditioned selective state-space temporal dynamics, sample-adaptive learnable wavelet spectral decomposition with cross-frequency attention, and dynamic-adjacency GCN over electrodes), fuse them, and align to frozen CLIP embeddings with contrastive loss plus repetition-consistency and prototype-level regularize

Key results

• Within-subject 200-way zero-shot: 54.8% Top-1 and 85.6% Top-5 average over 10 subjects, +3.9 Top-1 over UBP (50.9%).
• Cross-subject (leave-one-out): 15.3% Top-1 and 45.4% Top-5, +2.9 Top-1 over UBP (12.4%).
• Cross-session (first reported for this task): 40.8% Top-1 and 78.0% Top-5 average, vs 39.7% / 71.0% for UBP.

Surprising findings

• Removing the CPCA regularizer alone leaves average Top-1 nearly unchanged (54.6 vs 54.8), while removing RCR costs more (52.9).

Limitations

• Cross-subject accuracy remains low (15.3% Top-1), indicating limited subject-invariant generalization.
• Results reported from a single fixed random seed, with no variance estimates.
• Cross-session gains over UBP are small on average (+1.1 Top-1) and absent for some subjects (e.g., Sub-02, Sub-06).

Experimental setup

Datasets: THINGS-EEG

Benchmarks: THINGS-EEG

Metrics: Top-1 accuracy Top-5 accuracy

Baselines: BraVL NICE NICE-SA NICE-GA ATM-S UMind CognitionCapturer UBP

Builds on: Mamba (selective state-space model) CLIP NICE-EEG CLIP-ViT-H-14

Closest papers by meaning

• MindAlign: Bridging EEG, Vision, and Language for Zero-Shot Visual Decoding (NeurIPS 2026 · similarity 0.94)
• Visual Decoding and Reconstruction via EEG Embeddings with Guided Diffusion (NeurIPS 2024 · similarity 0.93)
• Decoding Natural Images from EEG for Object Recognition (ICLR 2024 · similarity 0.91)
• EEG2Video: Towards Decoding Dynamic Visual Perception from EEG Signals (NeurIPS 2024 · similarity 0.90)
• Neuro-3D: Towards 3D Visual Decoding from EEG Signals (CVPR 2025 · similarity 0.90)
• Human-Aligned Image Models Improve Visual Decoding from the Brain (ICML 2025 · similarity 0.89)