Research map / Architectures and efficiency / RNNs and normalization
Jacobian-Based Interpretation of Nonlinear Neural Encoding Model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes JNE, a Jacobian-dispersion interpretability metric for nonlinear neural encoding models, validated in simulations and applied to NSD fMRI to show a primary-to-higher-visual-cortex nonlinearity gradient.
Problem
Quantifying the nonlinearity of BOLD responses predicted by neural encoding models, since comparing linear vs nonlinear encoder R^2 fails when deep ANN features already embed nonlinearities.
Approach
JNE measures the across-sample standard deviation of L1-contracted centered Jacobians from stimulus representation to predicted voxel responses, with zero value for an ideal linear mapping; a forward-mode analytical Jacobian computation speeds it up over 99.9%.
Key results
- JNE is zero for the ideal linear simulation and increases with nonlinear activations, with GELU and Swish showing high JNE over broad input ranges while ReLU peaks only at x=0.
- Linear and nonlinear encoders achieve strongly correlated R^2 across ROIs, with minimum Pearson r = 0.82 in V1, so R^2 differences are uninformative about nonlinearity.
- JNE is lowest in V1 and highest in higher-order visual cortex (EBA, PPA, RSC, OPA, FFA), giving a primary < intermediate < higher-order gradient on NSD subject 1.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase the sample size used to estimate JNE (ablation) | JNE · JNE sample-size simulation | Pearson correlation with |sin(x)|: 0.87 -> 1.00 | Helped |
| Add activation functions to the otherwise linear network (ablation) | Three-layer ANN simulation · Controlled network-level simulation | JNE | Helped |
| Place activations in the deeper layer rather than the shallow layer (ablation) | Three-layer ANN simulation · Controlled network-level simulation | JNE | Helped |
| Add activations in both layers rather than only one layer (ablation) | Three-layer ANN simulation · Controlled network-level simulation | JNE | Helped |
| Vary input amplitude scale (ablation) | JNE · Uniform input distributions at three amplitude ranges | Mean JNE: 0.00 / 0.12 / 0.14 / 0.16 | No clear effect |
| Vary input distribution type and variance (ablation) | JNE · Uniform and normal input distributions | Mean JNE: 0.00 / 0.12 / 0.14 / 0.16 | No clear effect |
| Use L2 instead of L1 norm contraction (ablation) | JNE · NSD S1 significant visual cortex voxels | Pearson correlation between JNE maps: 1.00 | No clear effect |
| Z-score Jacobian entries before norm contraction (ablation) | JNE · NSD S1 significant visual cortex voxels | Pearson correlation between JNE maps: 0.95; 0.92; 0.94; 0.92; 0.99 | No clear effect |
| Use nonlinear rather than linear neural encoding (ablation) | CLIP-ViT neural encoding models · NSD S1 whole brain and visual ROIs | R² prediction performance | No clear effect |
| Use JNE to compare cortical regions along the visual hierarchy | JNE · NSD visual cortex: PVC, IVC, HVC | JNE | Helped |
| Use JNE-SS to measure stimulus-specific nonlinearity | JNE-SS · NSD image categories and visual ROIs | JNE-SS | Mixed |
Surprising findings
- Linear and nonlinear encoders have nearly identical R^2 patterns, suggesting CLIP features already encode the nonlinearity, making R^2-difference comparisons unreliable.
Limitations
- JNE reflects the composite nonlinearity of the input-to-BOLD pathway (including hemodynamics), not direct neural nonlinearity.
- Results are shown for a single encoding architecture and stimulus set; consistency across architectures and training objectives is untested.
- No absolute JNE threshold is defined, so only relative comparisons are supported.
Experimental setup
Datasets: Natural Scenes Dataset MS-COCO
Benchmarks: Natural Scenes Dataset
Metrics: JNE R^2 Pearson correlation noise ceiling Pearson r (simulation tracking)
Baselines: linear neural encoding model R^2 ratio between linear and nonlinear encoders
Builds on: CLIP Natural Scenes Dataset HCP-MMP atlas
Closest papers by meaning
- Characterizing control between interacting subsystems with deep Jacobian estimation (NeurIPS 2025 · similarity 0.84)
- SynBrain: Enhancing Visual-to-fMRI Synthesis via Probabilistic Representation Learning (NeurIPS 2025 · similarity 0.83)
- Not all solutions are created equal: An analytical dissociation of functional and representational similarity in deep linear neural networks (ICML 2025 · similarity 0.83)
- Functional Connectomes of Neural Networks (AAAI 2025 · similarity 0.83)
- Probabilistic Decomposed Linear Dynamical Systems for Robust Discovery of Latent Neural Dynamics (NeurIPS 2024 · similarity 0.83)
- CSBrain: A Cross-scale Spatiotemporal Brain Foundation Model for EEG Decoding (NeurIPS 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.