Research map / Reinforcement learning / Policy gradient and actor-critic

Signal-Adaptive Trust Regions for Gradient-Free Optimization of Recurrent Spiking Neural Networks

NeurIPS 2026 · Reinforcement learning › Policy gradient and actor-critic · approach: Alternative policy update rules · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes SATR, a KL trust-region rule whose budget scales with gradient signal energy, instantiated for Bernoulli-connectivity evolving-connectivity RSNN policies, plus a bitset kernel for faster binary rollouts.

Problem

Population-based gradient-free training of recurrent spiking neural network policies is fragile with finite populations because noisy gradient estimates induce overly large distributional updates, especially for near-deterministic Bernoulli connectivity.

Approach

Bound the KL divergence between successive sampling distributions by a budget proportional to the squared norm of the population gradient (signal energy), so the trust region expands under strong signal and contracts under noise; for factorized Bernoulli this yields the update rho += eta*sqrt(rho(1-rho))*g, and a bitset AND+popcount engine accelerates binary rollouts.

Key results

• On Humanoid with population 8192, SATR reaches 13,860 ± 1,107 return versus 12,242 ± 1,903 for EC and 12,544 ± 1,175 for ES.
• At population 128 on Humanoid, SATR retains 4,908 ± 32 while EC collapses to 765 ± 36.
• The bitset engine gives up to 2.74x faster rollouts on Humanoid (18,668 s vs 51,098 s at pop 8192).

Surprising findings

• Fixed-KL trust region (EC+TR) performance is highly sensitive to delta and does not transfer across population sizes.
• The paper reports SATR-RSNN outperforming PPO-LSTM at matched wall-clock budget on Humanoid, despite using gradient-free binary training.

Limitations

• Relies on local Fisher (second-order) KL approximation and a factorized distribution assumption.
• Energy results are analytical estimates from published Loihi per-operation figures, not measured on neuromorphic hardware.
• Results reported over only 3 random seeds.

Experimental setup

Datasets: Brax Humanoid Brax Hopper Brax Walker2d

Benchmarks: Brax Humanoid Brax Hopper Brax Walker2d

Metrics: undiscounted episodic return wall-clock training time energy estimate

Baselines: PPO-LSTM ES-RSNN EC-RSNN SG-RSNN (AbsBeta, SuperSpike, DSigmoid) EC+TR

Builds on: Evolving Connectivity TRPO evolution strategies PPO surrogate gradient

Closest papers by meaning

• Adaptive Surrogate Gradients for Sequential Reinforcement Learning in Spiking Neural Networks (NeurIPS 2025 · similarity 0.87)
• GRSN: Gated Recurrent Spiking Neurons for POMDPs and MARL (AAAI 2025 · similarity 0.87)
• Decision SpikeFormer: Spike-Driven Transformer for Decision Making (CVPR 2025 · similarity 0.86)
• Proxy Target: Bridging the Gap Between Discrete Spiking Neural Networks and Continuous Control (NeurIPS 2025 · similarity 0.86)
• Stay Hungry, Keep Learning: Sustainable Plasticity for Deep Reinforcement Learning (ICML 2025 · similarity 0.86)
• Trust Region Reward Optimization and Proximal Inverse Reward Optimization Algorithm (NeurIPS 2025 · similarity 0.86)