Research map / Architectures and efficiency / RNNs and normalization
Scaling Laws and Tradeoffs in Recurrent Networks of Expressive Neurons
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces ELM Networks of multi-timescale expressive leaky-memory neurons and studies how a fixed parameter budget should be split between neuron count, per-neuron complexity, and connectivity, with an information-theoretic model explaining the optimum.
Problem
How to allocate a fixed parameter budget between number of recurrent units, per-unit complexity, and per-unit connectivity; whether simple units are optimal in recurrent networks.
Approach
Treats per-unit complexity as a controllable scaling axis by building a recurrent layer from Expressive Leaky Memory neurons (multi-timescale memory, structured synaptic integration, nonlinear MLP), and frames the budget tradeoff with a closed-form effective representation information model with power-law SNR and spectral decay.
Key results
- On SHD-Adding, ELM Networks with roughly 150 neurons or 10 memory units approach the best previously reported accuracy of 83%.
- Under fixed budgets on Enwik8, an intermediate neuron complexity is optimal, and larger budgets favor more and more complex neurons (e.g., (N,d_m)=(512,15) shifting to (1024,25) when budget quadruples).
- A joint four-parameter information-theoretic fit qualitatively reproduces Enwik8 tradeoff curves across three ablations; a structured search yields the recipe d_m ~ sqrt(N_rec) with d_mlp=2d_m, d_tree=2d_mlp.
Surprising findings
- Performance is non-monotonic under fixed budget: a clear intermediate optimum in neuron count versus complexity emerges despite monotonic gains along each axis alone.
- Larger budgets favor both more and more complex neurons rather than only more neurons.
Limitations
- Hardware misalignment means equal parameter counts can differ in wall-time and memory, and parameter count is only a budget proxy.
- Theory abstracts away recurrence, topology and optimization; assumptions hold only approximately and the architecture-to-theory mapping is empirical.
- Structured Pareto search may miss optima outside its search space; noise-floor regime not yet reached at tested sizes.
Experimental setup
Datasets: SHD-Adding Spiking Heidelberg Digits Enwik8 NeuronIO
Benchmarks: SHD-Adding Enwik8 NeuronIO
Metrics: test accuracy test bits-per-character reducible MAE reducible error Pareto frontier
Baselines: Continuous Thought Machines
Builds on: Expressive Leaky Memory neuron SuperSpike surrogate gradient Transformer-XL preprocessing Continuous Thought Machines
Closest papers by meaning
- The Expressive Leaky Memory Neuron: an Efficient and Expressive Phenomenological Neuron Model Can Solve Long-Horizon Tasks. (ICLR 2024 · similarity 0.92)
- Latent Equilibrium: A unified learning theory for arbitrarily fast computation with arbitrarily slow neurons (NeurIPS 2021 · similarity 0.89)
- Learning better with Dale’s Law: A Spectral Perspective (NeurIPS 2023 · similarity 0.89)
- Emergent mechanisms for long timescales depend on training curriculum and affect performance in memory tasks (ICLR 2024 · similarity 0.88)
- Unconditional stability of a recurrent neural circuit implementing divisive normalization (NeurIPS 2024 · similarity 0.88)
- Structured flexibility in recurrent neural networks via neuromodulation (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.