Research map / Architectures and efficiency / RNNs and normalization
Information dynamics and Memory in Neural Networks through Fisher Information Diffusion
Haodong Qin, Tatyana Sharpee
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use Fisher-information-optimal block-structured initialization instead of Xavier, Kaiming, orthogonal, unitary | simple tanh RNN (100 neurons) · Copy task (Tdelay=50) | convergence speed and accuracy | Improved |
| use Fisher-optimal initialization with fixed recurrent weights, readout only | simple tanh RNN · Sequential MNIST | convergence and accuracy | Improved |
| use Fisher-optimal initialization with fixed recurrent weights | simple tanh RNN · CIFAR-10 and IndoorCVPR 09 sequential classification | convergence and accuracy | Improved |
| apply Fisher-optimal initialization to LSTM/GRU recurrent components | LSTM/GRU gates · sequential memory tasks | training performance | Improved |
| operate at criticality (spectral radius 1) alone (ablation) | two-population RNN · average Fisher information over 100 steps | retained Fisher information | Mixed |
| add sparse feedback links to feedforward chain (ablation) | chain RNN with 2-8 subpopulations · optimized Fisher information | Fisher information | Improved |
| add indiscriminate feedback (ablation) | chain RNN · optimized Fisher information | Fisher information | Hurt |
| increase number of subpopulations at fixed neuron count (ablation) | optimized chain RNN · total Fisher information | Fisher information | Improved |
| increase network size N | analytic Fisher diffusion vs simulation · Fisher information trajectories | mean-squared error | Improved |
Closest papers by meaning
- Dynamical phases of short-term memory mechanisms in RNNs (ICML 2025 · similarity 0.87)
- Why Diffusion Models Don’t Memorize: The Role of Implicit Dynamical Regularization in Training (NeurIPS 2025 · similarity 0.86)
- Complex priors and flexible inference in recurrent circuits with dendritic nonlinearities (ICLR 2024 · similarity 0.86)
- Hidden Traveling Waves bind Working Memory Variables in Recurrent Neural Networks (ICML 2024 · similarity 0.86)
- Learning dynamics in linear recurrent neural networks (ICML 2025 · similarity 0.86)
- Predictive Attractor Models (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
