Research map / Vision-language and multimodal / CLIP and zero-shot
ZEBRA: Towards Zero-Shot Cross-Subject Generalization for Universal Brain Visual Decoding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ZEBRA, a zero-shot fMRI-to-image decoding framework that disentangles subject-invariant and semantic-specific fMRI features via adversarial training with GRL and residual decomposition, generalizing to unseen subjects without fine-tuning.
Problem
Existing fMRI-to-image reconstruction methods need subject-specific models or fine-tuning, limiting scalability; zero-shot generalization to unseen subjects is largely unexplored.
Approach
Decompose fMRI embeddings residually into subject-invariant and subject-specific parts, use adversarial subject discriminators and a gradient reversal layer to make the invariant path subject-agnostic and the specific path semantic-rich, aligning semantic features to CLIP; only the invariant path is used at inference.
Key results
- Zero-shot ZEBRA reaches PixCorr 0.131 and SSIM 0.375 on average over subjects 1, 2, 5, 7, versus 0.057 and 0.297 for zero-shot NeuroPictor*.
- ZEBRA attains 74.6% AlexNet(2), 81.2% AlexNet(5), 72.2% Inception and 71.5% CLIP, outperforming zero-shot NeuroPictor* (71.4%, 74.7%, 62.5%, 66.0%).
- ZEBRA's SSIM of 0.384 on subject 1 (full model) is close to NeuroPictor's fully finetuned 0.375 average SSIM.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ZEBRA instead of NeuroPictor* for zero-shot decoding | NSD cross-subject reconstruction · NSD, zero-shot, subjects 1/2/5/7 | PixCorr: 0.057 -> 0.131 | Helped |
| Use ZEBRA instead of NeuroPictor* for zero-shot decoding | NSD cross-subject reconstruction · NSD, zero-shot, subjects 1/2/5/7 | SSIM: 0.297 -> 0.375 | Helped |
| Use ZEBRA instead of NeuroPictor* for zero-shot decoding | NSD cross-subject reconstruction · NSD, zero-shot, subjects 1/2/5/7 | Alex(5): 74.7% -> 81.2% | Helped |
| Use ZEBRA instead of NeuroPictor* for zero-shot decoding | NSD cross-subject reconstruction · NSD, zero-shot, subjects 1/2/5/7 | CLIP: 66.0% -> 71.5% | Helped |
| Add SIFE adversarial training (ablation) | ZEBRA · NSD, subject 1 | PixCorr: 0.089 -> 0.129 | Helped |
| Add the SIFE preservation anchor (ablation) | ZEBRA with SIFE adversarial training · NSD, subject 1 | PixCorr: 0.129 -> 0.134 | Helped |
| Add SSFE adversarial training (ablation) | ZEBRA with SIFE components · NSD, subject 1 | PixCorr: 0.134 -> 0.142 | Helped |
| Add the SSFE preservation anchor (ablation) | ZEBRA with SIFE and SSFE adversarial training · NSD, subject 1 | PixCorr: 0.142 -> 0.153 | Helped |
| Increase training subjects from 4 to 7 (ablation) | ZEBRA · NSD, subject 1 | PixCorr: 0.109 -> 0.153 | Helped |
| Increase training subjects from 4 to 7 (ablation) | ZEBRA · NSD, subject 1 | CLIP: 63.7% -> 72.3% | Helped |
Surprising findings
- Adding MindTuner-style decoding to NeuroPictor (the 'Our baseline') does not yield satisfactory zero-shot results, performing below NeuroPictor* on several metrics.
Limitations
- Semantic fidelity lags behind few-shot approaches, especially for rare object categories.
- Evaluated on a limited number of subjects (NSD), restricting demonstration of generalizability.
- Only image reconstruction studied; extension to video or text is not evaluated.
Experimental setup
Datasets: Natural Scenes Dataset MS-COCO UK Biobank
Benchmarks: Natural Scenes Dataset
Metrics: PixCorr SSIM AlexNet(2) AlexNet(5) Inception CLIP similarity EffNet-B SwAV
Baselines: Takagi et al. Ozcelik et al. MindEye1 UMBRAE NeuroPictor MindBridge MindEye2 MindTuner Our baseline
Builds on: MindEye2 NeuroPictor MindTuner fMRI-PTE BiMixCo
Closest papers by meaning
- See Through Their Minds: Learning Transferable Brain Decoding Models from Cross-Subject fMRI (AAAI 2025 · similarity 0.90)
- Toward Generalizing Visual Brain Decoding to Unseen Subjects (ICLR 2025 · similarity 0.89)
- $i$MIND: Insightful Multi-subject Invariant Neural Decoding (NeurIPS 2025 · similarity 0.88)
- Brain decoding: toward real-time reconstruction of visual perception (ICLR 2024 · similarity 0.88)
- Neuro-Vision to Language: Enhancing Brain Recording-based Visual Reconstruction and Language Interaction (NeurIPS 2024 · similarity 0.88)
- Multigranular Evaluation for Brain Visual Decoding (AAAI 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.