Research map / Label-efficient and robust learning / OOD and anomaly detection

Benchmarking Out-of-Distribution Generalization Capabilities of DNN-based Encoding Models for the Ventral Visual Cortex.

NeurIPS 2024 · Label-efficient and robust learning › OOD and anomaly detection · approach: Robust fine-tuning for OOD generalization · 3 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces MacaqueITBench, a macaque IT neural population dataset, and benchmarks how DNN-based linear encoding models generalize under attribute-based and cosine-distance OOD splits, finding that a closest-cosine-distance metric predicts the drop.

Problem

DNN-based encoding models of the ventral visual cortex are usually evaluated with random in-distribution splits, so their ability to generalize to out-of-distribution images under distribution shift is unknown.

Approach

Construct OOD train/test splits from image-computable attributes (hue, contrast, saturation, intensity, temperature) and cosine distance in DNN feature space, then show that a closest-cosine-distance metric between train and test images correlates with the drop in neural predictivity.

Key results

• Neural predictivity on OOD splits retains as little as 20% of in-distribution performance (worst case ViT on high-saturation split; best case CLIP at 0.66 for high-temperature).
• Closest Cosine Distance correlates with OOD performance drop at Spearman rho=-0.49 across 109 sessions in distance-based splits and -0.45 across attribute-based splits.
• Closest Cosine Distance outperforms MMD and covariate-shift metrics in predicting OOD performance.

Measured effects reported in the paper

Surprising findings

• Even low-level image attributes such as hue, contrast, and saturation cause large drops in neural predictivity for all eight pre-trained DNNs tested.

Limitations

• DNNs were not fine-tuned on neural data, so generalization could potentially be improved by training.
• Did not examine whether the images are OOD for the pre-trained DNNs themselves, since the images are naturalistic and may lie within pretraining distributions.
• Only multi-unit activity from 640 electrodes across seven monkeys, with analyses focused on trial-averaged responses.

Experimental setup

Datasets: MacaqueITBench

Benchmarks: MacaqueITBench

Metrics: ceiling-normalized squared Pearson correlation (r^2_pred/r^2 OOD/InD performance ratio Spearman correlation between distribution shift and performa

Baselines: MMD distance covariate shift distance

Builds on: ridge regression linear encoding models Maximum Mean Discrepancy covariate shift

Closest papers by meaning

• Generalization in data-driven models of primary visual cortex (ICLR 2021 · similarity 0.91)
• Performance-optimized deep neural networks are evolving into worse models of inferotemporal visual cortex (NeurIPS 2023 · similarity 0.89)
• Towards robust vision by multi-task learning on monkey visual cortex (NeurIPS 2021 · similarity 0.89)
• Neuron Activation Coverage: Rethinking Out-of-distribution Detection and Generalization (ICLR 2024 · similarity 0.88)
• Vision Transformer Neural Architecture Search for Out-of-Distribution Generalization: Benchmark and Insights (NeurIPS 2024 · similarity 0.87)
• Toward Generalizing Visual Brain Decoding to Unseen Subjects (ICLR 2025 · similarity 0.87)