Research map / Architectures and efficiency / RNNs and normalization
Unconditional stability of a recurrent neural circuit implementing divisive normalization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proves unconditional local stability of ORGaNICs (an oscillatory recurrent gated integrator circuit implementing dynamic divisive normalization) for identity recurrent weights, derives its energy function, conjectures stability for arbitrary recurrent weights, and trains it with naive BPTT on MNIST tasks.
Problem
Recurrent neurodynamical circuit models that implement divisive normalization are biologically plausible but hard to train because their nonlinear dynamics lack stability guarantees for high-dimensional systems, which makes them hard to train with BPTT.
Approach
Links dynamic divisive normalization in ORGaNICs to a system of coupled damped harmonic oscillators; using the indirect Lyapunov method and a Lyapunov-diagonal-stability/regular-splitting argument, it proves the normalization fixed point is unconditionally stable for identity recurrent weights, and derives an explicit energy function. The stability then enables vanilla BPTT training without gradie
Key results
- With identity recurrent weights the circuit is unconditionally locally (and globally linearized) asymptotically stable for all parameters and inputs, with an explicit energy function derived.
- On static MNIST ORGaNICs reach 98.1-98.2% test accuracy, versus 94.9-95.2% for SSN and 98.2% for an MLP.
- On sMNIST ORGaNICs reach 97.8% with 128 units (100k params) and 90.7% on psMNIST, comparable to LSTMs (97.3%, 92.6%) and trained without gradient clipping.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use ORGaNICs instead of SSN (80:20) | MNIST static classification · MNIST test | accuracy: 95.2% -> 98.2% | Helped |
| use ORGaNICs (80:80) instead of MLP (50) | MNIST static classification · MNIST test | accuracy: 98.2% -> 98.2% | No clear effect |
| use ORGaNICs (128 units) instead of LSTM (128 units) | sequential classification · sMNIST test | accuracy: 97.3% -> 97.8% | Helped |
| use ORGaNICs (128 units) instead of LSTM (128 units) | sequential classification · psMNIST test | accuracy: 92.6% -> 90.7% | Hurt |
| allow learned rather than fixed intrinsic time constants (ablation) | ORGaNICs (64 units) · sMNIST test | accuracy: 90.3% -> 97.7% | Helped |
| allow learned rather than fixed intrinsic time constants (ablation) | ORGaNICs (64 units) · psMNIST test | accuracy: 80.3% -> 89.9% | Helped |
| allow learned rather than fixed intrinsic time constants (ablation) | ORGaNICs (128 units) · sMNIST test | accuracy: 94.8% -> 97.8% | Helped |
| allow learned rather than fixed intrinsic time constants (ablation) | ORGaNICs (128 units) · psMNIST test | accuracy: 84.8% -> 90.7% | Helped |
Surprising findings
- Naive BPTT training without gradient clipping stays stable, whereas LSTMs typically need clipping.
- Stability was empirically observed for arbitrary recurrent weights with max singular value up to 2, but unstable at 3.
Limitations
- Unconditional stability proven only for identity recurrent weights and the 2D case; arbitrary recurrent weights are only conjectured.
- Euler discretization of the stiff continuous dynamics may cause numerical instabilities.
- Input-gain weight matrices are n x n, so parameter count grows quickly with hidden size.
Experimental setup
Datasets: MNIST sequential MNIST permuted sequential MNIST
Benchmarks: MNIST sequential MNIST permuted sequential MNIST
Metrics: test accuracy largest eigenvalue real part of Jacobian trajectory boundedness
Baselines: SSN MLP LSTM AntisymmetricRNN coRNN Lipschitz RNN
Builds on: ORGaNICs LSTM Lipschitz RNN coRNN stabilized supralinear network
Closest papers by meaning
- Scaling Laws and Tradeoffs in Recurrent Networks of Expressive Neurons (NeurIPS 2026 · similarity 0.88)
- Recurrent neural network dynamical systems for biological vision (NeurIPS 2024 · similarity 0.88)
- Learning Dynamics of RNNs in Closed-Loop Environments (NeurIPS 2025 · similarity 0.88)
- TIDE: Asymmetric Neural Circuits for Stabilized Temporal Inhibitory-Excitatory Dynamics (NeurIPS 2026 · similarity 0.87)
- Balanced Resonate-and-Fire Neurons (ICML 2024 · similarity 0.87)
- Partial observation can induce mechanistic mismatches in data-constrained models of neural dynamics (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.