Research map / Reinforcement learning / Visual and representation RL
Controlled maximal variability along with reliable performance in recurrent neural networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes NeuroMOP, which trains a feedforward value-function network to control input currents into a fixed random chaotic RNN so as to maximize future action-state entropy while avoiding terminal (absorbing) states, yielding high controlled neural variability with reliable task performance.
Problem
How neural variability can be maximized in neural networks without compromising behavioral/task performance, whereas prior work typically suppresses variability after learning to achieve high task performance.
Approach
Extends the maximum occupancy principle (MOP) from behavior to neural activity: an intrinsic, reward-free objective maximizing discounted future action entropy, with terminal states defined as absorbing dead states, so that the controller injects structured stochastic currents into a fixed random RNN and switches between stochastic and deterministic modes; the optimal policy is a Boltzmann over a
Key results
- In the energy-constraint task, NeuroMOP reaches the full episode length (T=1000) while showing larger average neural standard deviation than the epsilon-greedy R network, which suppresses activity to a low-energy silent regime.
- In the constrained neural space task, NeuroMOP confines two readout neurons within a square boundary while filling its interior, whereas the R network fails to occupy the region except at extremely small epsilon.
- Context-dependent NeuroMOP learns six shape constraints with a single network, with mean lifetimes reaching the training end of tend=5000 for some shapes and difficulty varying by shape.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| adapt action entropy to reduce stochasticity near the energy threshold | NeuroMOP · Energy constraint | lifetime | Helped |
| use state-dependent action selection rather than the R network’s fixed ε-greedy stochasticity | NeuroMOP versus R network · Constrained neural space | occupancy of available activity region | Helped |
| add context input to condition the value approximator on the required shape | NeuroMOP · Context-dependent neural space constraints | lifetime | Helped |
| add the approximated state-entropy term, increasing β (ablation) | NeuroMOP · Energy constraint | average standard deviation of activity jumps | Helped |
| add the approximated state-entropy term, increasing β (ablation) | NeuroMOP · Energy constraint | average standard deviation across trajectories | No clear effect |
| add the approximated state-entropy term with β = 1.5 (ablation) | NeuroMOP · Energy constraint | effective action dimensionality | Hurt |
| use noisy rather than deterministic RNN dynamics (ablation) | NeuroMOP · Energy constraint with noisy RNN | lifetime | Hurt |
| replace the saturating tanh transfer function with ReLU (ablation) | NeuroMOP · Energy constraint with ReLU RNN | lifetime | Helped |
| add extrinsic rewards to the MOP framework (ablation) | NeuroMOP · RNN behavior | variability | Hurt |
Surprising findings
- Reward-maximizing R networks can only avoid terminal states after extensive training and only by quenching their stochasticity, whereas MOP agents adapt entropy state-dependently.
- Introducing extrinsic reward into the MOP framework largely reduces network behavioral variability.
Limitations
- The policy is not learned; an oracle computes the exact optimal Boltzmann policy, and the computation of action selection remains costly.
- Only deterministic RNN dynamics with fixed weights are studied; spiking and Poisson-like variability is not addressed.
- Learning the structure of terminal states is not addressed.
Experimental setup
Benchmarks: energy constraint task constrained neural space (square boundaries) context-dependent shape constraints cartpole balancing (appendix)
Metrics: lifetime average standard deviation of activity effective action dimensionality state occupancy density
Baselines: R network (epsilon-greedy reward maximizer) free (uncontrolled) RNN
Builds on: maximum occupancy principle soft Q-learning / maximum entropy RL epsilon-greedy exploration
Closest papers by meaning
- BLEND: Behavior-guided Neural Population Dynamics Modeling via Privileged Knowledge Distillation (ICLR 2025 · similarity 0.85)
- Structured flexibility in recurrent neural networks via neuromodulation (NeurIPS 2024 · similarity 0.85)
- Measuring and Controlling Solution Degeneracy across Task-Trained Recurrent Neural Networks (NeurIPS 2025 · similarity 0.85)
- Feedback control guides credit assignment in recurrent neural networks (NeurIPS 2024 · similarity 0.85)
- Measure gradients, not activations! Enhancing neuronal activity in deep reinforcement learning (NeurIPS 2025 · similarity 0.84)
- Sufficient conditions for offline reactivation in recurrent neural networks (ICLR 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.