Research map / Architectures and efficiency / RNNs and normalization
Measuring and Controlling Solution Degeneracy across Task-Trained Recurrent Neural Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds a framework to quantify and control solution degeneracy in task-trained vanilla RNNs at behavior, dynamics, and weight levels, sweeping task complexity, feature learning (muP), width, and low-rank/L1 regularization over 3,400 RNNs.
Problem
Task-trained RNNs that reach similar loss can learn strikingly different internal solutions (solution degeneracy), and the factors governing this variability across behavior, neural dynamics, and weights are not systematically understood.
Approach
Unified three-level degeneracy metrics (OOD loss std, DSA dynamical distance, permutation-invariant Frobenius weight distance) applied to a large controlled sweep, empirically validating the Contravariance Principle and revealing contravariant (task complexity, feature learning) versus covariant (width, regularization) effects between dynamics and weights.
Key results
- Higher task complexity lowered dynamical and behavioral degeneracy but raised weight degeneracy (contravariant effect), consistent across tasks.
- Larger networks and low-rank or L1 regularization reduced degeneracy at dynamics, weight, and behavior levels (covariant effect).
- Stronger feature learning (larger muP gamma) reduced dynamical degeneracy and increased weight and behavioral degeneracy.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase task complexity | Task-trained RNNs across four neuroscience tasks · Four neuroscience tasks | dynamical degeneracy | Helped |
| Increase task complexity | Task-trained RNNs across four neuroscience tasks · Four neuroscience tasks | weight degeneracy | Hurt |
| Increase task complexity | Task-trained RNNs across four neuroscience tasks · Temporal generalization | behavioral degeneracy | Helped |
| Increase memory demand by lengthening the delay | Delayed Discrimination RNNs · Delayed Discrimination | dynamical degeneracy | Helped |
| Increase memory demand by lengthening the delay | Delayed Discrimination RNNs · Delayed Discrimination | weight degeneracy | Hurt |
| Increase memory demand by lengthening the delay | Delayed Discrimination RNNs · Temporal generalization | behavioral degeneracy | Helped |
| Add an auxiliary output for the magnitude of the stimulus difference | Delayed Discrimination RNNs · Delayed Discrimination | dynamical degeneracy | Helped |
| Add an auxiliary output for the magnitude of the stimulus difference | Delayed Discrimination RNNs · Delayed Discrimination | weight degeneracy | Hurt |
| Add an auxiliary output for the magnitude of the stimulus difference | Delayed Discrimination RNNs · Temporal generalization | behavioral degeneracy | Helped |
| Increase feature-learning strength using higher γ | μP RNNs across four neuroscience tasks · Four neuroscience tasks | dynamical degeneracy | Helped |
| Increase feature-learning strength using higher γ | μP RNNs across four neuroscience tasks · Four neuroscience tasks | weight degeneracy | Hurt |
| Increase feature-learning strength using higher γ | μP RNNs across four neuroscience tasks · Temporal generalization | behavioral degeneracy | Hurt |
Surprising findings
- Wider standard RNNs can become lazier and show higher dynamical degeneracy, so width effects must be isolated with muP.
- Stronger feature learning reduced dynamical degeneracy on the training task but increased OOD behavioral variability, decoupling dynamics from behavior.
Limitations
- Considers networks equivalent only when they reach similar training loss; does not study tasks with multiple distinct solutions.
- Dynamical and behavioral degeneracy decouple, and the mechanistic link is not established.
- SVCCA representational degeneracy does not follow the same factor trends, and the GRU experiment is only a preliminary test on one task.
Experimental setup
Datasets: N-Bit Flip-Flop Delayed Discrimination Sine Wave Generation Path Integration Lorenz 96
Benchmarks: N-Bit Flip-Flop Delayed Discrimination Sine Wave Generation Path Integration Lorenz 96
Metrics: DSA distance (dynamical degeneracy) permutation-invariant Frobenius distance (weight degeneracy) std of OOD MSE under temporal generalization (behavioral deg SVCCA distance (representational degeneracy) weight-change norm neural tangent kernel alignment
Baselines: SVCCA
Builds on: Contravariance Principle Dynamical Similarity Analysis Maximal Update Parameterization SVCCA
Closest papers by meaning
- Learning dynamics in linear recurrent neural networks (ICML 2025 · similarity 0.91)
- Learning Dynamics of RNNs in Closed-Loop Environments (NeurIPS 2025 · similarity 0.89)
- Emergent mechanisms for long timescales depend on training curriculum and affect performance in memory tasks (ICLR 2024 · similarity 0.89)
- Dynamical phases of short-term memory mechanisms in RNNs (ICML 2025 · similarity 0.88)
- Structured flexibility in recurrent neural networks via neuromodulation (NeurIPS 2024 · similarity 0.87)
- Recurrent neural networks: vanishing and exploding gradients are not the end of the story (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.