Research map / Probabilistic and scientific modeling / Dynamical systems
Range, not Independence, Drives Modularity in Biologically Inspired Representations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Derives necessary and sufficient conditions on the support of source variables under which energy-efficient nonnegative linear autoencoders modularise, showing range (rectangular support) rather than statistical independence drives modularity, and validates predictions in nonlinear feedforward/recurrent networks and entorhinal grid-cell/reward data.
Problem
Determining when biologically inspired (nonnegative, energy-efficient) networks modularise their representation of source variables versus mixing them, beyond the restrictive case of statistically independent sources.
Approach
Shows that modularisation of optimal nonnegative, L2-efficient linear autoencoders depends on the range (extremal support shape) of sources rather than statistical independence; gives necessary and sufficient conditions equivalent to whether the convex hull of the data encloses a data-derived ellipse.
Key results
- Theory predicts modularity of linear autoencoders from source support; the degree to which inequalities are broken tracks mixing, while statistical dependence measures (source multiinformation) do not predict modularisation.
- In nonlinear feedforward and RNN experiments, cutting corners from source support degrades modularity much more than changing NSI while preserving rectangular support.
- Linear RNNs trained on sinusoids modularise for irrational frequency ratios and mix for harmonic ratios.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| remove corner regions from the source support | nonlinear feedforward networks · What-Where Regression | modularity | Hurt |
| introduce source correlations while preserving rectangular support instead of cutting corners | nonlinear feedforward networks · What-Where Regression | modularity | No clear effect |
| remove corners from the source support | nonlinear autoencoder · nonlinear autoencoding of sources | modularity | Hurt |
| introduce range-independent correlations instead of cutting corners | nonlinear autoencoder · nonlinear autoencoding of sources | modularity | No clear effect |
| remove corners from the source support | QLAE · Isaac3D subset | modularity | Hurt |
| preserve corner points while progressively breaking statistical independence | student RNN · teacher-student distillation trajectories | modularity | No clear effect |
| remove corner points from the source support | student RNN · teacher-student distillation trajectories | modularity | Hurt |
| use harmonically related frequencies rather than irrational frequency ratios | linear RNN · sinusoidal regression | modularity | Hurt |
| use irrational frequency ratios rather than harmonic ratios | nonlinear RNN · mixed sinusoidal regression | modularity | Helped |
| include rooms with randomly scattered rewards so reward and position remain range-independent | linear RNN · space and reward representation | modularity | Helped |
| use fixed reward positions in every room, making reward and position range-dependent | linear RNN · space and reward representation | modularity | Hurt |
Surprising findings
- Only the support of the source distribution matters, even for very low probability regions, rather than statistical independence.
- Statistically dependent but range-independent variables (e.g., reward and position in random rooms) can still modularise, while range-dependent variables mix even when only weakly correlated.
Limitations
- Theory covers 1-D sources, L2 activity norm and perfect reconstruction; multidimensional and other norms only partly addressed.
- Conditions are sufficient in some settings and necessary only in the linear autoencoder case; nonlinear extensions are empirical.
Experimental setup
Datasets: Isaac3D
Benchmarks: Isaac3D
Metrics: conditional InfoM (CInfoM) normalised source multiinformation (NSI) maximum mixing angle of neuron weight vectors
Baselines: source multiinformation (statistical independence measure) QLAE
Builds on: efficient coding hypothesis quantised latent autoencoding (QLAE) InfoM modularity metric independent components analysis
Closest papers by meaning
- Neural networks are more modular than single neurons suggest (NeurIPS 2026 · similarity 0.89)
- Learning to live with Dale's principle: ANNs with separate excitatory and inhibitory units (ICLR 2021 · similarity 0.86)
- Breaking Neural Network Scaling Laws with Modularity (ICLR 2025 · similarity 0.85)
- Shaping the distribution of neural responses with interneurons in a recurrent circuit model (NeurIPS 2024 · similarity 0.85)
- The impact of allocation strategies in subset learning on the expressive power of neural networks (ICLR 2025 · similarity 0.85)
- Scaling Laws and Tradeoffs in Recurrent Networks of Expressive Neurons (NeurIPS 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.