Research map / Architectures and efficiency / RNNs and normalization

Leveraging Low-Rank and Sparse Recurrent Connectivity for Robust Closed-Loop Control

ICLR 2024 · Architectures and efficiency › RNNs and normalization · approach: Brain-inspired recurrent architectures · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Parameterizes RNN recurrent weights as a random-sparse, low-rank product and shows that, for closed-form continuous-time networks (CfCs), low-rank sparse connectivity improves closed-loop robustness under distribution shift on Arcade Learning Environment and MuJoCo imitation tasks.

Problem

Recurrent agents fit offline on expert demonstrations often fail when deployed closed-loop under distribution shift; it is unclear how recurrent connectivity structure (rank and sparsity) affects this robustness.

Approach

Parameterize recurrent weights as a rank-r factorization multiplied elementwise by a fixed random sparsity mask at initialization, and analyze how rank and sparsity modulate spectral radius, spectral norm, singular value decay and state-space dimensionality. Low-rank connectivity is found to improve robustness under distribution shift for CfCs, whereas sparsity helps LSTMs in-distribution.

Key results

• Under distribution shift, low-rank, low-sparsity CfCs are the best performing models in Seaquest, and low-rank models perform on par with or better than higher-rank ones across architectures.
• In-distribution, LSTMs tolerate high sparsity well, whereas CfCs and RNNs degrade at high sparsity relative to LSTMs.
• CfCs learn lower recurrent spectral norms and lower-dimensional state-space trajectories than LSTMs and RNNs.

Measured effects reported in the paper

Surprising findings

• Sparsity and low-rank pruning have opposing effects: high sparsity hurts robustness under distribution shift while low rank improves it.
• Extending the rank/sparsity parameterization to input weights was found to be detrimental to performance (results not shown).

Limitations

• Theoretical results on spectral radius and norm are proven only for some initialization schemes; other cases are empirical.
• Spectral analyses are used as proxies for robustness and comparisons across architectures are acknowledged as potentially limited.
• Input weights are not parameterized by rank and sparsity, since doing so hurt performance.

Experimental setup

Datasets: Seaquest expert trajectories (from DQN) Alien expert trajectories (from DQN) HalfCheetah expert trajectories (from DQN)

Benchmarks: Arcade Learning Environment Seaquest Arcade Learning Environment Alien MuJoCo HalfCheetah

Metrics: normalized online reward in-distribution normalized reward averaged across distribution shifts spectral radius spectral norm singular value decay effective dimensionality (PCA)

Baselines: RNN LSTM GRU CNN full-rank fully-connected CfC

Builds on: closed-form continuous-time neural networks low-rank RNN echo state networks LSTM

Closest papers by meaning

• Learning Dynamics of RNNs in Closed-Loop Environments (NeurIPS 2025 · similarity 0.87)
• Feedback control guides credit assignment in recurrent neural networks (NeurIPS 2024 · similarity 0.87)
• Learning Reveals Invisible Structure in Low-Rank RNNs (NeurIPS 2026 · similarity 0.86)
• How connectivity structure shapes rich and lazy learning in neural circuits (ICLR 2024 · similarity 0.85)
• Inferring stochastic low-rank recurrent neural networks from neural data (NeurIPS 2024 · similarity 0.85)
• Brain network science modelling of sparse neural networks enables Transformers and LLMs to perform as fully connected (NeurIPS 2025 · similarity 0.85)