Research map / Architectures and efficiency / RNNs and normalization
Associative memory and dead neurons
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that the Krotov-Hopfield energy function has flat, non-compact regions caused by dead neurons, analyzes the resulting sensitivity and stability problems, and proposes modified dynamics R(u)(g(Wu+b)-u) with Lyapunov functions lacking flat directions.
Problem
The Lyapunov (energy) function of the Krotov-Hopfield large associative memory model becomes flat along non-compact regions wherever neurons die (activation saturation or symmetry), so it cannot determine stability, steady states, or basins of attraction there.
Approach
Characterizes dead neurons as flat energy directions in the kernel of the activation Jacobian, shows the stability of a steady state can be analyzed on the Hessian range, and modifies the dynamics by a matrix R(u) so that a family of parametric energy functions (E2, E3 with a matrix S) has no dead-neuron flat directions.
Key results
- Shows that dead neurons create non-compact flat energy regions in the Krotov-Hopfield energy, so the energy is independent of the dead-neuron coordinates.
- Proves that for a Gaussian orthogonal ensemble weight matrix, E||W_perp||_2 = sqrt(1 - k/N) E||W||_2 when k neurons are dead, showing a stabilizing effect.
- Derives a parametric family of Lyapunov functions with no flat directions from dead neurons when beta or gamma is nonzero, including for non-symmetric weights.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the modified associative-memory dynamics with E2 or E3 instead of the original energy formulation | Associative memory · Associative-memory state space | flat energy directions corresponding to dead neurons | Helped |
Surprising findings
- The Krotov-Hopfield energy is flat over non-compact regions at every point where neurons die, so energy alone cannot certify stability there.
- Energy-based analysis of the old model's stability can be recovered from the Hessian range, and zero Hessian modes do not destabilize the dynamics.
Limitations
- Stability analysis of the flat-region case requires knowledge of the Hessian range, which may be hard to compute for realistic architectures.
- Modified dynamics require choosing R(u) and S, and the matrix inequality is nonlinear in general.
- Results are mostly theoretical with only illustrative numerical figures.
Experimental setup
Metrics: energy value stability of steady states eigenvalue spectrum of Hessian-based matrix expected spectral norm ratio
Baselines: Krotov-Hopfield energy function (Lagrangian-based) Hopfield 1984 energy function
Builds on: Krotov and Hopfield large associative memory model Dense Associative Memory Modern Hopfield network LaSalle invariance principle
Closest papers by meaning
- Back to the Continuous Attractor (NeurIPS 2024 · similarity 0.86)
- Exponential Dynamic Energy Network for High Capacity Sequence Memory (NeurIPS 2025 · similarity 0.84)
- Unconditional stability of a recurrent neural circuit implementing divisive normalization (NeurIPS 2024 · similarity 0.84)
- Stability Analysis of Switched Linear Systems with Neural Lyapunov Functions (AAAI 2024 · similarity 0.84)
- Dense Associative Memory Through the Lens of Random Features (NeurIPS 2024 · similarity 0.83)
- Factual recall in linear associative memories: sharp asymptotics and mechanistic insights (NeurIPS 2026 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.