Research map / Vision-language and multimodal / Human and embodied modeling

FlatClip: Reusing Image Foundation Models for fMRI Representation Learning via Cortical Flatmaps

NeurIPS 2026 · Vision-language and multimodal › Human and embodied modeling · approach: fMRI visual decoding

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Renders cortical fMRI activity as flatmap image sequences and trains only a lightweight MLP probe on frozen SigLIP2 features, testing whether cortical geometry helps image-pretrained encoders transfer to fMRI prediction.

Problem

Choosing the spatial scale of fMRI representation (ROI, voxel, or surface) is a central design decision for fMRI foundation models, and it is unclear how well a frozen image encoder can use the spatial organization of cortical activity for fMRI prediction.

Approach

Render subject-level cortical activity as anatomy-linked flatmap frames (via pycortex on fsLR surfaces), encode each frame with a frozen SigLIP2 encoder, average-pool over time, and train only a small MLP probe; spatial-perturbation controls test whether the cortical arrangement matters.

Key results

• On HCP sex classification FlatClip reaches 82.06 ACC / 82.04 wF1, above ROI-level baselines (e.g., Brain-LM 64.12) but below SwiFT (86.03) and Omni-fMRI (92.86).
• On NSD COCO80, the NSDgeneral input region gives mean mAP 16.347 versus 12.085 for SigLIP2-Cortex and above the voxel-level baselines SwiFT, NeuroSTORM and Omni-fMRI (mean mAP 9.374, 8.188, 4.360).
• Block-shuffling, cortical-pixel permutation and within-hemisphere shuffle all reduce HCP sex-classification accuracy (to 76.14, 72.93 and 70.90 respectively) relative to real flatmaps (82.06).

Surprising findings

• Direct-volume slice encoding with frozen SigLIP2 reached 89.32 wF1 on HCP, above the flatmap result of 82.04, which the authors note is at a higher image-count cost.
• A randomly initialized SigLIP2 encoder with the same flatmap input still reached 70.56 ACC on HCP, showing much of the performance comes from the input format and architecture.

Limitations

• Flatmaps introduce cuts and metric distortion and omit subcortical and brainstem signals, which likely hurts PPMI performance.
• Temporal mean pooling discards frame order, so temporal dynamics are not modeled.
• FlatClip remains below the strongest voxel-level models (e.g., Omni-fMRI, SwiFT) on HCP and several other resting-state tasks.

Experimental setup

Datasets: HCP PPMI ADNI NSD COCO

Benchmarks: HCP sex classification PPMI PD diagnosis ADNI MCI vs CN ADNI AD vs CN NSD COCO80 multi-label recognition

Metrics: accuracy weighted F1 mAP balanced accuracy macro F1

Baselines: BrainLM BrainMASS LCM Brain-Harmony SwiFT NeuroSTORM Omni-fMRI CortexMAE-F DINOv2 ViT-B

Builds on: SigLIP2 pycortex Brain-Harmony CortexMAE-F

Closest papers by meaning

• NeuroClips: Towards High-fidelity and Smooth fMRI-to-Video Reconstruction (NeurIPS 2024 · similarity 0.88)
• Platonic Representations in the Human Brain: Unsupervised Recovery of Universal Geometry (NeurIPS 2026 · similarity 0.88)
• Meta-Learning an In-Context Transformer Model of Human Higher Visual Cortex (NeurIPS 2025 · similarity 0.87)
• MindEye2: Shared-Subject Models Enable fMRI-To-Image With 1 Hour of Data (ICML 2024 · similarity 0.87)
• SIM: Surface-based fMRI Analysis for Inter-Subject Multimodal Decoding from Movie-Watching Experiments (ICLR 2025 · similarity 0.87)
• CLIP-MSM: A Multi-Semantic Mapping Brain Representation for Human High-Level Visual Cortex (AAAI 2025 · similarity 0.86)