Research map / Probabilistic and scientific modeling / Dynamical systems
Slow Transition to Low-Dimensional Chaos in Heavy-Tailed Recurrent Neural Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that finite-size RNNs with Lévy alpha-stable recurrent weights have a sharp quiescent-to-chaos transition at a predicted gain g* (decaying ~ (ln N)^(-1/alpha)), that heavier tails give a slower transition to chaos but lower Lyapunov and participation-ratio dimensionality.
Problem
Infinite-size mean-field theory predicts that tanh RNNs with Lévy alpha-stable (heavy-tailed) weights are always chaotic, which conflicts with biologically observed heavy-tailed synaptic weights and leaves finite-size behavior, edge-of-chaos robustness, and dimensionality of such networks poorly und
Approach
Derives a finite-size critical gain g* = exp(-<Xi_{N,alpha}>) from an annealed Kesten-process analysis of quiescent-state stability for alpha-stable weights, validates it numerically, and shows heavier tails widen the edge-of-chaos gain regime while compressing attractor dimension (Lyapunov and participation ratio).
Key results
- Finite-size theory predicts critical gain g* decaying as 1/(ln N)^(1/alpha) for alpha<2, while Gaussian g* quickly converges to 1/sqrt(2) (mean-field).
- Heavy-tailed RNNs (alpha=1) show a slower MLE rise with gain than Gaussian networks, keeping near-edge-of-chaos dynamics over a broader gain range; the transition location shifts left with N, more so for heavier tails.
- Heavier tails lower the Lyapunov dimension and participation ratio near the edge of chaos, with the Lyapunov dimension drop being larger than the PR drop.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use heavier-tailed weights (lower α) instead of Gaussian weights | RNNs near the transition to chaos · Autonomous RNNs | maximum Lyapunov exponent transition width | Helped |
| use heavier-tailed weights (lower α) instead of Gaussian weights | RNNs near chaos · Autonomous RNNs | Lyapunov dimension | Hurt |
| use heavier-tailed weights (lower α) instead of Gaussian weights | RNNs near chaos · Autonomous RNNs | participation ratio | Hurt |
| use heavier-tailed weights (lower α) instead of Gaussian weights | RNNs near chaos · Noisy stimulus-driven RNNs | maximum Lyapunov exponent transition width | Helped |
| use heavier-tailed weights (lower α) instead of Gaussian weights | RNNs near chaos · Noisy stimulus-driven RNNs | participation ratio | Mixed |
| use α = 0.5 weights | RNN · Autonomous and noisy dynamics | maximum Lyapunov exponent | Hurt |
| remove bottom 95% of recurrent weights (ablation) | Heavy-tailed RNN · Quenched RNN | critical gain | No clear effect |
| retain top outlier weights rather than prune them (ablation) | Heavy-tailed RNN · Quenched RNN | critical gain | Helped |
| remove top outlier weights (ablation) | Heavy-tailed RNN · Chaotic regime | Lyapunov dimension | Helped |
| increase network size N | Lévy α-stable RNN · Transition to chaos | critical gain | Hurt |
| increase accumulation duration or simulation length (ablation) | RNN Lyapunov-exponent estimates · Autonomous and noisy RNNs | reported dynamics trends | No clear effect |
| increase network size, accumulation duration, or simulation length (ablation) | RNN Lyapunov spectra and dimensionality estimates · Autonomous and noisy RNNs | reported dimensionality trends | No clear effect |
Surprising findings
- For alpha=0.5, networks do not transition to chaos; the MLE becomes negative again at large gain.
- Removing the top 1% of weights (mega-synapses) eliminates the slow transition, indicating robustness stems from extreme outliers.
Limitations
- Analysis is for untrained, rate-based networks with homogeneous tanh units and no Dale's law or spiking.
- Quenched transition location varies across realizations and lacks analytical treatment; for alpha<1 the MLE becomes non-monotonic and no chaos transition is observed.
- Reservoir-computing results are only a proof-of-concept on a single delayed XOR task.
Experimental setup
Benchmarks: delayed XOR reservoir computing task
Metrics: maximal Lyapunov exponent critical gain g* Lyapunov dimension participation ratio fraction of small final-state components
Baselines: Gaussian RNN (alpha=2)
Builds on: Kesten process annealed approximation Lyapunov exponent QR algorithm Kaplan-Yorke dimension
Closest papers by meaning
- Mechanistic Interpretability of RNNs emulating Hidden Markov Models (NeurIPS 2025 · similarity 0.86)
- Unconditional stability of a recurrent neural circuit implementing divisive normalization (NeurIPS 2024 · similarity 0.86)
- Back to the Continuous Attractor (NeurIPS 2024 · similarity 0.85)
- Inference of Neural Dynamics Using Switching Recurrent Neural Networks (NeurIPS 2024 · similarity 0.85)
- Uniform-in-Time Weak Propagation of Chaos in Shallow Neural Networks (NeurIPS 2026 · similarity 0.85)
- Dynamical phases of short-term memory mechanisms in RNNs (ICML 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.