Research map / Architectures and efficiency / RNNs and normalization

Neural networks are more modular than single neurons suggest

NeurIPS 2026 · Architectures and efficiency › RNNs and normalization · approach: Mechanistic circuit interpretability

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces a nested hierarchy (unit, orthogonal, linear) of modularity and learns subspace decompositions via task-sufficiency or contrastive minimal-pair objectives, showing that networks lacking unit-level modularity often show clear subspace-level double dissociations across RNNs, vision models, and language models.

Problem

Whether neural networks with mixed selectivity (no neurons specialized for single functions) are modular; prior work concluded absence of unit-level specialization implies non-modularity, and selectivity-based proxies do not test causal dissociation.

Approach

Define a nested hierarchy of modularity (permutation ⊂ orthogonal ⊂ invertible) and learn linear decompositions of hidden representations into functionally specific subspaces, evaluated causally by selective subspace ablation (double dissociation) or by retrieval selectivity and brain-encoding alignment.

Key results

• In tanh multitask RNNs, unit-level DDI is 0.00 while orthogonal DDI is 0.37 and linear DDI is 0.47 on the Anti vs DM1 pair.
• For Swin Base, invertible decomposition gives content/style selectivity ratios R_s=16.1 and R_c=5.3 versus near-chance (~1.1, ~1.0) for permutation.
• For Pythia 410M, invertible decomposition yields top-1 syntax selectivity S1=0.94 and top-1 semantic selectivity S2=0.66.

Surprising findings

• Tanh RNNs that look non-modular at unit level show clear double dissociations after subspace decomposition.
• Content and style subspaces align with visual cortex in a hierarchical pattern: style predicts early visual areas, content predicts higher areas.

Limitations

• Decomposition learning is non-convex and sensitive to hyperparameters such as subspace dimensionality and condition number bound.
• Vision and language evaluation relies on retrieval-based selectivity rather than behavioral or causal ablation.
• Residual subspace content remains unclear.

Experimental setup

Datasets: MS-COCO Paint by Numbers Stylized ImageNet procedure MSR Paraphrase Corpus Pereira 2018 Natural Scenes Dataset Yang multitask cognitive tasks Mante task

Benchmarks: Mante task Yang multitask cognitive tasks content-style retrieval (MS-COCO/Paint by Numbers) syntax-semantics retrieval (Pereira 2018) Natural Scenes Dataset fMRI encoding Pereira fMRI encoding

Metrics: double dissociation index (DDI) fractional task variance top-k retrieval accuracy content/style selectivity ratio syntax/semantics selectivity brain encoding predictivity

Baselines: unit-level decomposition Sinkhorn permutation orthogonal decomposition

Builds on: Yang et al. 2019 multitask RNNs Mante et al. 2013 context-dependent integration Stylized ImageNet GeoTorch

Closest papers by meaning

• Breaking Neural Network Scaling Laws with Modularity (ICLR 2025 · similarity 0.90)
• Range, not Independence, Drives Modularity in Biologically Inspired Representations (ICLR 2025 · similarity 0.89)
• Not all solutions are created equal: An analytical dissociation of functional and representational similarity in deep linear neural networks (ICML 2025 · similarity 0.87)
• Geometry of naturalistic object representations in recurrent neural network models of working memory (NeurIPS 2024 · similarity 0.86)
• Flexible Context-Driven Sensory Processing in Dynamical Vision Models (NeurIPS 2024 · similarity 0.86)
• Uncovering a Universal Abstract Algorithm for Modular Addition in Neural Networks (NeurIPS 2025 · similarity 0.85)