Research map / Reinforcement learning / MDPs and planning
GRSN: Gated Recurrent Spiking Neurons for POMDPs and MARL
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a temporal alignment paradigm (TAP) that maps one spiking-neuron update to one RL state transition, plus gated recurrent spiking neurons (GRSN), to run spiking RL on POMDPs and SMAC with RNN-level performance at about half the estimated energy.
Problem
Current spiking reinforcement learning methods use multiple SNN time steps to decode a single RL decision, a temporal mismatch that wastes compute and underuses SNN temporal dynamics, especially for partially observable tasks.
Approach
Align each single-step spiking neuron update with one MDP state transition so the SNN acts like an RNN over the episode, and add gated recurrent connections (forget and input gates) on the input current and learnable decay with soft reset to strengthen memory.
Key results
- On SMAC 8m, GRSN-QMIX reached 99.4% win rate versus 97.6% for GRU-QMIX, and on 27m_vs_30m 96.4% versus 79.4%.
- On Pendulum-P, GRSN with TAP reached -195.8 mean reward and 64 time steps versus -1069.9 for LIF without TAP using 256 steps.
- Estimated energy for GRSN-SNN is about 49.1% lower than GRU-RNN on PO control and SMAC tasks, per the paper's SOP-based estimate.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add GRSN gated recurrent neurons to LIF under TAP (ablation) | Spiking RL · Pendulum-P | mean reward: -824.8 -> -195.8 | Helped |
| Add GRSN gated recurrent neurons to LIF under TAP (ablation) | QMIX · SMAC-8m | mean reward: 15.3 -> 19.9 | Helped |
| Use TAP rather than repeated input and rate coding (ablation) | LIF spiking RL · Pendulum-P | mean reward: -1069.9 -> -824.8 | Helped |
| Use TAP rather than repeated input and rate coding (ablation) | GRSN spiking RL · Pendulum-P | mean reward: -235.4 -> -195.8 | Helped |
| Use TAP rather than repeated input and rate coding (ablation) | QMIX-LIF · SMAC-8m | mean reward: 16.9 -> 15.3 | Hurt |
| Use TAP rather than repeated input and rate coding (ablation) | QMIX-GRSN · SMAC-8m | mean reward: 19.7 -> 19.9 | Helped |
| Replace GRU with GRSN | QMIX · SMAC-2s3z | mean reward: 19.9 ± 0.18 -> 19.9 ± 0.20 | No clear effect |
| Replace GRU with GRSN | QMIX · SMAC-3s_vs_5z | win rate (%): 97.1 ± 3.46 -> 93.0 ± 5.48 | Hurt |
| Replace GRU with GRSN | QMIX · SMAC-27m_vs_30m | win rate (%): 79.4 ± 12.19 -> 96.4 ± 3.74 | Helped |
| Replace GRU with GRSN | QMIX · SMAC-8m | win rate (%): 97.6 ± 3.49 -> 99.4 ± 1.48 | Helped |
| Replace GRU RNN with GRSN SNN | PO-Ctrl. energy estimate · PO-Ctrl. | energy (KpJ): 628.82 -> 7.92 | Helped |
| Use GRSN rather than LIF (ablation) | Spiking RL · Pendulum-P | training time (h): 3.55 -> 3.42 | Helped |
Surprising findings
- Reported energy savings are computed from SOP counts with assumed per-operation energy costs rather than measured on neuromorphic hardware.
- The paper reports that the GRSN convergence rate is slightly weaker than GRU in some environments despite similar final performance.
Limitations
- Energy consumption is estimated analytically (SOP counting), not measured on hardware.
- Experiments are limited to small classic control tasks and SMAC maps; some results have high variance (e.g., LIF with TAP on CartPole-P).
Experimental setup
Datasets: CartPole Pendulum StarCraft Multi-Agent Challenge (SMAC)
Benchmarks: CartPole-V CartPole-P Pendulum-V Pendulum-P SMAC 8m SMAC 2s3z SMAC 8m_vs_9m SMAC 3s_vs_5z SMAC 27m_vs_30m SMAC MMM2
Metrics: mean episode reward win rate (%) standard deviation training time synaptic operations (SOP) energy (pJ)
Baselines: GRU-based RNN-RL MLP actor-critic LIF-based SNN with TAP QMIX-GRU LIF with repeated input and rate coding (T=4)
Builds on: QMIX SAC TD3 LIF neuron surrogate gradient
Closest papers by meaning
- Temporal Flexibility in Spiking Neural Networks: Towards Generalization Across Time Steps and Deployment Friendliness (ICLR 2025 · similarity 0.89)
- Proxy Target: Bridging the Gap Between Discrete Spiking Neural Networks and Continuous Control (NeurIPS 2025 · similarity 0.88)
- Decision SpikeFormer: Spike-Driven Transformer for Decision Making (CVPR 2025 · similarity 0.88)
- Temporal Misalignment in ANN-SNN Conversion and its Mitigation via Probabilistic Spiking Neurons (ICML 2025 · similarity 0.88)
- SpikingSSMs: Learning Long Sequences with Sparse and Parallel Spiking State Space Models (AAAI 2025 · similarity 0.88)
- Adaptive Surrogate Gradients for Sequential Reinforcement Learning in Spiking Neural Networks (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.