Research map / Vision-language and multimodal / CLIP and zero-shot
Bridging the Semantic Latent Space between Brain and Machine: Similarity Is All You Need
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes BrainSem, a self-supervised fMRI-to-CLIP joint embedding model trained with a fuzzy one-to-many matching loss and virtual fMRI augmentation, evaluated on zero-shot neural decoding of the GOD dataset.
Problem
Aligning fMRI brain responses with a semantic embedding space to decode unseen visual categories from limited fMRI-image pairs, and testing whether brain and CLIP-like networks share similar semantic (locality-sensitive) representations.
Approach
Treats fMRI-to-image matching as a fuzzy one-to-many contrastive problem where all ImageNet images of the same class serve as positives, combined with a neuro-semantic reorganization that creates virtual fMRI responses as convex combinations of real recordings.
Key results
- BrainSem averages 25.6% Top-1 and 53.6% Top-5 zero-shot accuracy on GOD across five subjects, versus 14.0% Top-1 for BraVL (the prior SOTA).
- Ablation: using both virtual brain responses and the visual semantic library raises Top-1 from 18.8% to 25.6%.
- Swapping CLIP image encoder variants changes Top-1 only between 24.8% and 25.8%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use BrainSem instead of BraVL | zero-shot neural classification · GOD test, average | Top-1 accuracy: 14.0% -> 25.6% | Helped |
| use BrainSem instead of BraVL | zero-shot neural classification · GOD test, average | Top-5 accuracy: 53.1% -> 53.6% | Helped |
| use CLIP ViT-L/14 instead of ViT-B/16 as image encoder (ablation) | BrainSem · GOD test | Top-1 accuracy: 25.6% -> 25.8% | Helped |
| use CLIP ViT-L/14 instead of ViT-B/16 as image encoder (ablation) | BrainSem · GOD test | Top-5 accuracy: 53.6% -> 54.4% | Helped |
| add visual semantic library (ablation) | BrainSem with ViT 6 layers, batch 100 · GOD test | Top-1 accuracy: 19.4% -> 21.0% | Helped |
| add virtual brain activity (ablation) | BrainSem with ViT 6 layers, batch 100 · GOD test | Top-1 accuracy: 19.4% -> 24.2% | Helped |
| add visual semantic library and virtual brain activity (ablation) | BrainSem with ViT 24 layers, batch 200 · GOD test | Top-1 accuracy: 18.8% -> 25.6% | Helped |
| add visual semantic library and virtual brain activity (ablation) | BrainSem with ViT 12 layers, batch 100 · GOD test | Top-5 accuracy: 46.2% -> 58.4% | Helped |
| use CLIP ViT-B/32 instead of ViT-B/16 as image encoder (ablation) | BrainSem · GOD test | Top-1 accuracy: 25.6% -> 25.2% | Hurt |
| use CLIP ResNet50 instead of ViT-B/16 as image encoder (ablation) | BrainSem · GOD test | Top-1 accuracy: 25.6% -> 24.8% | Hurt |
| use CLIP ViT-B/32 instead of ViT-B/16 as image encoder (ablation) | BrainSem · GOD test | Top-5 accuracy: 53.6% -> 52.0% | Hurt |
Surprising findings
- Raw fMRI and CLIP features show weaker locality-sensitive structure than BrainSem embeddings in ISOMAP/t-SNE visualizations.
- A smaller fMRI encoder outperforms a larger one when trained only on raw annotation data, while the fuzzy matching makes performance barely sensitive to encoder size.
Limitations
- Decoding is performed per subject; group-level decoding remains difficult due to individual variability.
- Misclassification occurs mainly when test samples have large semantic gaps from known classes.
- Only the GOD dataset is evaluated, with a small number of fMRI recordings.
Experimental setup
Datasets: GOD (Horikawa and Kamitani 2017) ImageNet
Benchmarks: GOD zero-shot neural classification ImageNet-1k retrieval subset
Metrics: Top-1 accuracy Top-5 accuracy
Baselines: CADA-VAE MVAE MMVAE MoPoE-VAE BraVL
Builds on: CLIP contrastive learning mixup BraVL
Closest papers by meaning
- Finding Shared Decodable Concepts and their Negations in the Brain (ICLR 2025 · similarity 0.88)
- The Semantic Hub Hypothesis: Language Models Share Semantic Representations Across Languages and Modalities (ICLR 2025 · similarity 0.87)
- CLIP-MSM: A Multi-Semantic Mapping Brain Representation for Human High-Level Visual Cortex (AAAI 2025 · similarity 0.86)
- In Silico Mapping of Visual Categorical Selectivity Across the Whole Brain (NeurIPS 2025 · similarity 0.86)
- Far from the Shallow: Brain-Predictive Reasoning Embedding through Residual Disentanglement (NeurIPS 2025 · similarity 0.86)
- BrainSCUBA: Fine-Grained Natural Language Captions of Visual Cortex Selectivity (ICLR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.