Research map / Architectures and efficiency / RNNs and normalization
The Expressive Leaky Memory Neuron: an Efficient and Expressive Phenomenological Neuron Model Can Solve Long-Horizon Tasks.
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces the Expressive Leaky Memory (ELM) neuron, a recurrent cell with multiple learnable-timescale leaky memory units and an MLP for nonlinear synaptic integration, which fits a biophysical L5 pyramidal neuron model with few parameters and solves long-range tasks (LRA, SHD-Adding).
Problem
Replicating the spike-level input/output relationship of a detailed biophysical cortical neuron model with a small, interpretable surrogate, and assessing whether such a neuron-inspired recurrent cell can handle long-horizon temporal tasks.
Approach
A recurrent cell whose memory units decay with explicit, learnable timescales and are updated by an MLP applied to synaptic traces (optionally after branch-wise summation), allowing expressive nonlinear dendritic-like integration with far fewer parameters than TCNs or LSTMs.
Key results
- ELM reaches the 0.991 spike AUC threshold on NeuronIO with about 50K parameters, versus about 10M for the prior TCN and 266K for LSTM; Branch-ELM reaches 0.9915 AUC with 5329 parameters.
- On LRA, ELM scores 77.29% on Pathfinder-X (16k length) where Transformer and Longformer fail, and 71.15% on Pathfinder, outperforming Chrono-LSTM on most tasks.
- On SHD-Adding, ELM outperforms LSTM (186K vs 956K params), especially at fine bin sizes, where LSTM suffers from vanishing gradients.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ELM instead of Chrono-LSTM | Long Range Arena · Pathfinder-X | accuracy: FAIL$^*$ -> 77.29 | Helped |
| Use ELM instead of Transformer | Long Range Arena · Pathfinder-X | accuracy: FAIL -> 77.29 | Helped |
| Increase ELM memory units from d_m=10 to d_m=200 (ablation) | ELM · Pathfinder | accuracy: 57.83 -> 72.98 | Helped |
| Increase ELM memory units from d_m=200 to d_m=300 (ablation) | ELM · Pathfinder | accuracy: 72.98 -> 71.85 | Hurt |
| Use dropout 0.3 instead of dropout 0 (ablation) | ELM · Pathfinder | test accuracy: 0.7275 -> 0.823 | Helped |
| Increase dropout from 0.3 to 0.5 (ablation) | ELM · Pathfinder | test accuracy: 0.823 -> 0.8063 | Hurt |
| Add nonlinear MLP synaptic integration and multiple memory units (ablation) | ELM · NeuronIO | AUC: 0.9117 -> 0.9918 | Helped |
| Use Branch-ELM with ordered synaptic inputs and learnable synapses (ablation) | ELM · NeuronIO | AUC: 0.9918 ± 0.0002 -> 0.9924 ± 0.0002 | Helped |
| Remove the hidden layer or reduce integration complexity (ablation) | ELM · NeuronIO | prediction performance | Hurt |
| Allow memory timescales to reach at least 25 ms (ablation) | ELM · NeuronIO | prediction performance | Helped |
| Use constant memory timescales around 25 ms (ablation) | ELM · NeuronIO | prediction performance | No clear effect |
| Increase lambda to allow more rapid memory updates (ablation) | ELM · NeuronIO | prediction performance | Helped |
Surprising findings
- A single-memory-unit ELM with linear integration fits LIF neurons, and a small ELM fits the neuron better than classic LIF/ALIF, which fail to reach the 0.991 AUC threshold.
- Transformer and Longformer completely fail on Pathfinder-X, while the ELM neuron reliably solves it.
Limitations
- Trained with BPTT, which is not biologically plausible; mechanistic interpretation of parameters is limited.
- Rudimentary dendrites (no apical/basal distinction) and reliance on oversampling synaptic inputs.
- Single ELM neuron falls short of deep SOTA models (S4, Mega) on LRA; large ELM sizes on later tasks.
Experimental setup
Datasets: NeuronIO Spiking Heidelberg Digits (SHD) SHD-Adding Long Range Arena (Image, Pathfinder, Pathfinder-X, ListOps, Text, Retrieval)
Benchmarks: NeuronIO SHD SHD-Adding Long Range Arena
Metrics: AUC RMSE accuracy number of trainable parameters FLOPs
Baselines: Temporal convolutional network (TCN) LSTM Chrono-LSTM Transformer Longformer S4 Mega S5 LIF ALIF Spiking neural network (LIF)
Builds on: LSTM Chrono-LSTM Leaky-RNN S5 Long Range Arena
Closest papers by meaning
- Scaling Laws and Tradeoffs in Recurrent Networks of Expressive Neurons (NeurIPS 2026 · similarity 0.92)
- TC-LIF: A Two-Compartment Spiking Neuron Model for Long-Term Sequential Modelling (AAAI 2024 · similarity 0.88)
- CLIF: Complementary Leaky Integrate-and-Fire Neuron for Spiking Neural Networks (ICML 2024 · similarity 0.87)
- The Dormant Spiking Neuron: A State-Driven Mechanism for Efficient Spiking Neural Networks (NeurIPS 2026 · similarity 0.87)
- HetSyn: Versatile Timescale Integration in Spiking Neural Networks via Heterogeneous Synapses (NeurIPS 2025 · similarity 0.87)
- Autaptic Synaptic Circuit Enhances Spatio-temporal Predictive Learning of Spiking Neural Networks (ICML 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.