Research map / Architectures and efficiency / RNNs and normalization
Isotropic Activation Functions Enable Deindividuated Neurons and Adaptive Topologies
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes isotropic (orthogonal-equivariant) activation functions that let SVD diagonalise dense layers into one-to-one connections, enabling function-invariant neurogenesis/neurodegeneration, 50% sparsification, and adaptive-width MLPs.
Problem
Adapting the topology (adding/removing neurons) of dense networks without degrading function is hard because elementwise activations individuate neurons along an arbitrary basis, making connectivity dense and neuron removal disruptive.
Approach
Replace permutation-equivariant elementwise activations with orthogonally equivariant isotropic activations (f(Rx)=Rf(x)); then SVD of the middle affine map can be applied function-invariantly, yielding a diagonal one-to-one layer where neurons can be appended (scaffold neurons) or pruned by small singular value, with a 'intrinsic length' parameter absorbing residual bias.
Key results
- Neurogenesis and diagonalisation preserve logits to machine precision (epsilon ~1e-4 logit L2 distance); neurodegeneration from width 100 to 25 with trainable compensatory parameters gives epsilon 0.09 without retraining.
- Adaptive-width isotropic MLPs on a changing MNIST/FMNIST/EMNIST task schedule reach test accuracies of 93.3% (MNIST), 85.1% (FMNIST), 86.1% (A-J EMNIST) versus control 92.9%, 84.7%, 85.0% at T3.
- Epoch-wise diagonalisation/neurogenesis took 29 ms and neurodegeneration 39 ms versus 19.0 s training per epoch on CIFAR-10.
Surprising findings
- Adaptive networks showed a bifurcation in H2/H3 widths during the T2 task schedule, with some runs converging to a stable merged width rather than continuing to grow.
- Orthogonal transforms break the reparameterisation symmetry during training (optimiser coupling), so the invariance holds at inference but not training time.
Limitations
- Experiments restricted to MLPs with tanh as placeholder isotropic activation; claims not tested on transformers or convnets.
- Neurodegeneration is only approximately function-preserving, with error growing for large pruning fractions; new scaffold neurons need careful symmetry-breaking initialisation to avoid stalled gradients.
Experimental setup
Datasets: CIFAR-10 MNIST FMNIST EMNIST
Benchmarks: CIFAR-10 MNIST FMNIST EMNIST
Metrics: test accuracy mean per-example L2 logit distance sum of singular values
Baselines: static isotropic MLP control
Builds on: tanh Adam BatchNorm LayerNorm RMSNorm
Closest papers by meaning
- How connectivity structure shapes rich and lazy learning in neural circuits (ICLR 2024 · similarity 0.85)
- Shaping the distribution of neural responses with interneurons in a recurrent circuit model (NeurIPS 2024 · similarity 0.85)
- Improving equilibrium propagation without weight symmetry through Jacobian homeostasis (ICLR 2024 · similarity 0.84)
- Scaling Laws and Tradeoffs in Recurrent Networks of Expressive Neurons (NeurIPS 2026 · similarity 0.84)
- Neuron Platonic Intrinsic Representation From Dynamics Using Contrastive Learning (ICLR 2025 · similarity 0.84)
- Two Tales of Single-Phase Contrastive Hebbian Learning (ICML 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.