Research map / Architectures and efficiency / RNNs and normalization
Exponential Dynamic Energy Network for High Capacity Sequence Memory
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes EDEN, a dynamic energy network with a slow modulatory population and softmax-based fast population, showing analytic escape times and exponential sequence memory capacity O(γ^N) versus linear for prior sequence networks.
Problem
Energy-based (Hopfield-type) memory networks retrieve single static memories and have linear capacity; sequence memory models with energy guarantees and high capacity are lacking.
Approach
Extends the exponential-capacity static energy network with a slow delay population that asymmetrically modulates the energy surface over time, so short-timescale energy functions govern local dynamics and memory transitions, with analytic escape times and capacity bounds.
Key results
- Analytic escape time <T_d/T_f> = -(T_d/T_f) ln(1 - sqrt(alpha_s/alpha_c)) matches simulations with mean absolute error 5.96 T_f units.
- EDEN capacity scales as O(gamma^N) with gamma > 2, versus O(N) for the linear reference network, simulated for N from 10 to 35.
- Phase transition at alpha_s/alpha_c = 1 separates static memory (infinite escape time) from dynamic sequence transitions.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use EDEN's softmax hidden layer and slow asymmetric interactions instead of the reference network's linear interactions | EDEN versus reference network · Sequence memory capacity | capacity scaling: O(N) -> O(\gamma^N) | Helped |
| Use analytical escape times computed from the energy function | EDEN · EDEN escape time | mean absolute error: 5.96\mathcal{T}_f | Helped |
Surprising findings
- The paper reports that the fast population's energy function can be locally characterized even though no single global energy governs the dynamics.
- Authors claim the successive bifurcations in prior work are due to changing energy-landscape stability, and that capacity gains in prior work may stem from separating memory contributions.
Limitations
- Assumes fixed synaptic strengths with no learning during training.
- Assumes Rademacher binary patterns and strictly separated timescales, with Markovian transitions.
- Capacity analysis relies on mean-field and Gaussian approximations and is evaluated only at the phase boundary.
Experimental setup
Datasets: MNIST
Metrics: fixed point capacity average escape time bit error probability
Baselines: linear-capacity multiple-timescale sequence network (reference network)
Builds on: Hopfield networks dense associative memory exponential capacity energy networks
Closest papers by meaning
- Dynamical phases of short-term memory mechanisms in RNNs (ICML 2025 · similarity 0.87)
- The Dormant Spiking Neuron: A State-Driven Mechanism for Efficient Spiking Neural Networks (NeurIPS 2026 · similarity 0.87)
- Recurrent Self-Attention Dynamics: An Energy-Agnostic Perspective from Jacobians (NeurIPS 2025 · similarity 0.87)
- Are Conventional SNNs Really Efficient? A Perspective from Network Quantization (CVPR 2024 · similarity 0.86)
- Unconditional stability of a recurrent neural circuit implementing divisive normalization (NeurIPS 2024 · similarity 0.86)
- Predictive Attractor Models (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.