Research map / Reinforcement learning / Policy gradient and actor-critic
Proxy Target: Bridging the Gap Between Discrete Spiking Neural Networks and Continuous Control
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a proxy target network (continuous ANN imitating the online spiking actor via an implicit output-space imitation loss) to enable smooth Polyak target updates for SNN actors in TD3, letting LIF-based SNNs match or exceed ANN performance on MuJoCo control tasks.
Problem
Target network soft updates assume smooth output changes, which discrete, non-differentiable spiking neurons violate, destabilizing and degrading SNN actor training in off-policy continuous control RL.
Approach
Replace the spiking target actor with a continuous ANN proxy trained each update to imitate the online SNN actor's outputs via an MSE loss (implicit output-space update), so target outputs change smoothly; the proxy is discarded at deployment, leaving the SNN inference unchanged.
Key results
- PT-LIF achieves an APG of 5.84% over the ANN (TD3) baseline across five MuJoCo tasks, with HalfCheetah 10103 ± 607 vs ANN 10857 ± 475 and Ant 5383 ± 250 vs 4770 ± 1014.
- Vanilla LIF SNN reaches an APG of -10.54% relative to the ANN; the proxy target raises the LIF SNN's APG by about 32% on the LIF setting (Appendix table: 32.15%).
- PT-LIF average inference energy is 11.09 nJ versus 12.21 nJ for vanilla LIF and 281.78 nJ for the ANN TD3 actor.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace the spiking target actor with a continuous proxy actor and implicitly align it to the online SNN | TD3 with LIF neurons · MuJoCo environments (average) | average performance gain (APG) over ANN (TD3): 32.15% | Helped |
| Replace the spiking target actor with a continuous proxy actor and implicitly align it to the online SNN | TD3 with CLIF neurons · MuJoCo environments (average) | average performance gain (APG) over ANN (TD3): 15.03% | Helped |
| Replace the spiking target actor with a continuous proxy actor and implicitly align it to the online SNN | TD3 with dynamic neurons · MuJoCo environments (average) | average performance gain (APG) over ANN (TD3): 4.87% | Helped |
| Use the proxy target framework instead of vanilla Actor-Critic (ablation) | LIF spiking actor on MuJoCo · MuJoCo environments (average) | average performance gain (APG): 32.15% | Helped |
| Use the proxy target framework instead of vanilla Actor-Critic (ablation) | CLIF spiking actor on MuJoCo · MuJoCo environments (average) | average performance gain (APG): 15.03% | Helped |
| Use the proxy target framework instead of vanilla Actor-Critic (ablation) | Dynamic-neuron spiking actor on MuJoCo · MuJoCo environments (average) | average performance gain (APG): 4.87% | Helped |
| Add the proxy target framework (ablation) | ANN (TD3) · MuJoCo environments (average) | average performance gain (APG): -8.38% | Hurt |
| Use the proxy target framework instead of vanilla LIF Actor-Critic (ablation) | LIF spiking actor · Hopper-v4 | max average return: 3520 ± 94 -> 3385 ± 157 | Hurt |
| Use the proxy target framework instead of vanilla LIF Actor-Critic (ablation) | LIF spiking actor · Ant-v4 | max average return: 4294 ± 1170 -> 5383 ± 250 | Helped |
| Use the proxy target framework instead of vanilla LIF Actor-Critic (ablation) | LIF spiking actor · HalfCheetah-v4 | max average return: 9404 ± 625 -> 10103 ± 607 | Helped |
| Use the proxy target framework instead of vanilla LIF Actor-Critic (ablation) | LIF spiking actor · Walker2d-v4 | max average return: 1862 ± 1450 -> 4314 ± 423 | Helped |
| Use the proxy target framework instead of vanilla CLIF Actor-Critic (ablation) | CLIF spiking actor · MuJoCo environments | max average return | Mixed |
Surprising findings
- The proxy target framework does not improve the ANN itself (PN-ANN APG -8.38%), suggesting gains are specific to SNN dynamics.
- The simplest LIF neuron achieves the best overall performance under the proxy framework, contrary to prior findings favoring complex neuron models.
Limitations
- Results remain at simulation level; not deployed on neuromorphic or edge hardware.
- Evaluated only on five MuJoCo tasks with TD3 as the base algorithm.
- Proxy network is wider than the SNN and adds training-time cost (K=3 extra proxy updates per step).
Experimental setup
Datasets: MuJoCo IDP-v4 Ant-v4 HalfCheetah-v4 Hopper-v4 Walker2d-v4
Benchmarks: InvertedDoublePendulum-v4 Ant-v4 HalfCheetah-v4 Hopper-v4 Walker2d-v4
Metrics: max average return over 5 seeds average performance gain (APG) variance across seeds energy per inference (nJ) firing rate
Baselines: ANN (TD3) ANN-SNN conversion pop-SAN MDC-SAN ILC-SAN Vanilla LIF
Builds on: TD3 STBP Polyak soft target update CLIF neurons dynamic neurons
Closest papers by meaning
- Adaptive Surrogate Gradients for Sequential Reinforcement Learning in Spiking Neural Networks (NeurIPS 2025 · similarity 0.89)
- GRSN: Gated Recurrent Spiking Neurons for POMDPs and MARL (AAAI 2025 · similarity 0.88)
- Temporal Flexibility in Spiking Neural Networks: Towards Generalization Across Time Steps and Deployment Friendliness (ICLR 2025 · similarity 0.87)
- Sign Gradient Descent-based Neuronal Dynamics: ANN-to-SNN Conversion Beyond ReLU Network (ICML 2024 · similarity 0.87)
- Are Conventional SNNs Really Efficient? A Perspective from Network Quantization (CVPR 2024 · similarity 0.87)
- The Dormant Spiking Neuron: A State-Driven Mechanism for Efficient Spiking Neural Networks (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.