Research map / Continual learning / Continual fine-tuning of LLMs

Brain Decodes Deep Nets

CVPR 2024 · Continual learning › Continual fine-tuning of LLMs · approach: Synthetic data and merging for CLIP · 10 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Maps pre-trained vision transformers onto fMRI brain voxels via a factorized, topology-smooth brain encoding model (FactorTopy) to visualize deep-network layers, scales and channels against brain ROIs.

Problem

How to explicitly map deep vision network features (space, layer, scale, channel) to brain regions so that the internal computation of pre-trained models can be visualized and compared with the visual brain hierarchy.

Approach

Use a brain encoding model with factorized soft selectors over space, layer, and scale plus a topology-based smoothness constraint across voxels, then read the learned layer selectors as a brain-to-network map, and use brain hierarchy knowledge to analyze networks (training objective, scaling, fine-tuning).

Key results

• CLIP shows the best layer-to-brain hierarchical alignment; its hierarchy slope rises with scale (0.32 for base to 0.50 for large, 0.52 for XL) while other models' slopes decrease when scaled.
• After fine-tuning on small datasets, CLIP and MAE keep brain score closer to original (CLIP 0.131 to 0.115 on ISIC) than SAM (0.111 to 0.086) and DiNOv2 (0.128 to 0.085).
• Channel clustering of brain voxels with 20 clusters aligns with known ROIs; V1 features capture edge information and FFA features are face-selective.

Measured effects reported in the paper

Surprising findings

• Supervised ImageNet and SAM final layers align with mid-level brain regions rather than high-level ones.
• DiNOv2 and SAM lose hierarchical structure rapidly when fine-tuned on small datasets.

Limitations

• Relies on NSD, the only publicly available large paired image-fMRI dataset.
• Brain-network comparison is less informative if networks compute in non-brain-like ways.
• Analysis of hierarchy uses only coarse ROIs and ignores feedback computation.

Experimental setup

Datasets: Natural Scenes Dataset (NSD) ISIC EuroSAT

Benchmarks: Natural Scenes Dataset (NSD) brain encoding

Metrics: R2 brain score hierarchy slope b0 b1 root sum squared difference

Baselines: Layer-specific spatial feature selection (state-of-the-art brain encoding [1]) Per-voxel independent encoding

Builds on: Memory Encoding Model (Algonauts 2023 winner) Natural Scenes Dataset

Closest papers by meaning

• A Convolutional Neural Network Interpretable Framework for Human Ventral Visual Pathway Representation (AAAI 2024 · similarity 0.87)
• MoRE-Brain: Routed Mixture of Experts for Interpretable and Generalizable Cross-Subject fMRI Visual Decoding (NeurIPS 2025 · similarity 0.87)
• Toward Generalizing Visual Brain Decoding to Unseen Subjects (ICLR 2025 · similarity 0.87)
• See Through Their Minds: Learning Transferable Brain Decoding Models from Cross-Subject fMRI (AAAI 2025 · similarity 0.86)
• CLIP-MSM: A Multi-Semantic Mapping Brain Representation for Human High-Level Visual Cortex (AAAI 2025 · similarity 0.86)
• Transformer brain encoders explain human high-level visual responses (NeurIPS 2025 · similarity 0.86)