Research map / Reinforcement learning / Policy gradient and actor-critic
Adaptive Surrogate Gradients for Sequential Reinforcement Learning in Spiking Neural Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Analyzes surrogate-gradient slope effects in deep SNNs and proposes TD3BC+JSRL, which bootstraps sequence-based online RL for a spiking Crazyflie drone controller using a privileged guiding policy and an adaptive slope schedule.
Problem
Training spiking neural networks for continuous control with RL is hampered by poorly understood surrogate gradient slope effects and by the inability to collect long sequences needed to bridge the warm-up period of stateful SNNs, since early crashes terminate episodes.
Approach
Combine a privileged, non-spiking guiding policy (JumpStart RL) that fills the replay buffer during warm-up with online TD3 training of the spiking actor plus an exponentially decaying BC term, and schedule the surrogate gradient slope adaptively from reward.
Key results
- TD3BC+JSRL reaches an average return of about 400 under reward curriculum, while BC, TD3BC and TD3 stay at or below -200.
- Shallower or scheduled surrogate slopes give a x2.1 improvement in RL training and final performance; the adaptive/interval schedule reduces epochs to reach reward 100 by x4.5 versus fixed slope 100.
- On the real Crazyflie, the SNN without action history achieves 0.04 m xy position error versus 0.1 m for the best ANN with action history and 0.25 m for an ANN without action history, which failed to stabilize the drone.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use TD3BC+JSRL instead of offline BC, TD3BC, or vanilla TD3 | SNN drone controller · Crazyflie position control with reward curriculum | average reward: 400 points; other methods fail to exceed –200 points | Helped |
| Use adaptive or scheduled surrogate slopes rather than a fixed slope of 100 | SNN online RL training · Crazyflie control | training efficiency: ×4.5 | Helped |
| Use shallower or scheduled surrogate slopes rather than steep slopes | SNN online RL · Crazyflie control | training and final deployed performance: ×2.1 | Helped |
| Vary surrogate slope settings | SNN behavioral cloning · Crazyflie control, supervised learning | final performance | No clear effect |
| Add a BC term to the method with jump-start (ablation) | TD3BC+JSRL · Crazyflie control | average final reward: $334 \pm 25.59$ -> $412 \pm 6.72$ | Helped |
| Add a BC term to the method with jump-start (ablation) | TD3BC+JSRL · Crazyflie control | steps to reward 100: $33030 \pm 1600$ -> $2200 \pm 124$ | Helped |
| Add a jump-start period (ablation) | TD3BC+JSRL with BC term · Crazyflie control | average final reward: $32 \pm 1.2$ -> $412 \pm 6.72$ | Helped |
| Add both the BC term and jump-start period (ablation) | TD3BC+JSRL · Crazyflie control | average final reward: $36 \pm 4.1$ -> $412 \pm 6.72$ | Helped |
| Add temporal processing without explicit action history | SNN controller · Crazyflie position control | position error [m]: 0.1 -> 0.04 | Helped |
| Use the SNN without action history rather than the ANN with action history | Crazyflie trajectory tracking controller · Figure-eight and square trajectories | trajectory error [m]: 0.21 -> 0.24 | Hurt |
Surprising findings
- The ANN without action history could not stabilize the quadrotor, while the SNN with only the latest state did.
- Shallow surrogate slopes reduce gradient-direction alignment toward zero in deeper layers, yet this helps RL exploration while hurting supervised convergence.
Limitations
- Relies on the availability of a guiding policy that must produce stable early behavior.
- Real-world SNN controller oscillates and has reduced reliability compared with the ANN baseline; trajectory error (0.24 m) is worse than the ANN with action history (0.21 m).
- Energy estimate is analytical, not measured on neuromorphic hardware.
Experimental setup
Datasets: simulated Crazyflie rollouts from guiding policy and spiking policy
Benchmarks: Crazyflie quadrotor position control (simulation) real Crazyflie 2.1 hover and trajectory tracking NeuroBench
Metrics: average return steps to reward 100 position error (m) trajectory error (m) footprint (kb) activation sparsity SynOps
Baselines: BC TD3BC TD3 ANN with action history (Eschmann et al.) ANN without action history
Builds on: JSRL TD3 TD3BC fast sigmoid surrogate gradient R2D2
Closest papers by meaning
- NDOT: Neuronal Dynamics-based Online Training for Spiking Neural Networks (ICML 2024 · similarity 0.89)
- Proxy Target: Bridging the Gap Between Discrete Spiking Neural Networks and Continuous Control (NeurIPS 2025 · similarity 0.89)
- Sign Gradient Descent-based Neuronal Dynamics: ANN-to-SNN Conversion Beyond ReLU Network (ICML 2024 · similarity 0.88)
- Decision SpikeFormer: Spike-Driven Transformer for Decision Making (CVPR 2025 · similarity 0.88)
- Take A Shortcut Back: Mitigating the Gradient Vanishing for Training Spiking Neural Networks (NeurIPS 2024 · similarity 0.88)
- Temporal Flexibility in Spiking Neural Networks: Towards Generalization Across Time Steps and Deployment Friendliness (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.