Research map / Architectures and efficiency / Spiking networks
Autonomous Driving with Spiking Neural Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces SAD, an end-to-end spiking neural network that unifies perception, prediction, and planning for autonomous driving on nuScenes, trading some accuracy for large estimated energy savings.
Problem
Autonomous driving systems need integrated perception, prediction, and planning under strict energy budgets, but SNNs have not been shown effective for complex multi-subtask real-world vision applications like driving.
Approach
A fully spiking end-to-end driving stack: a spiking token mixer encoder (sequence repetition, pretrained on ImageNet), spiking decoder with sequential alignment producing BEV outputs, a dual-pathway spiking prediction module with a Gaussian uncertainty latent, and a spiking GRU refining sampled trajectories, all trained in three stages.
Key results
- Perception: SAD reaches 35.62% mean BEV IoU versus 42.69% for ST-P3 and 28.19% for VED, outperforming VED by 7.43 points.
- Prediction: SAD gets 32.74 future IoU and 20.00 PQ, below ST-P3 (38.63 IoU, 31.72 PQ), so it lags ANN methods.
- Energy: SAD's estimated planning energy is 46.92 mJ, about 7.33x lower than Freespace (344.11 mJ) and 75.03x lower than ST-P3 (3520.40 mJ).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace the ANN baseline with SAD | nuScenes perception · nuScenes | Avg. IoU: 42.69 -> 35.62 | Hurt |
| Replace the ANN baseline with SAD | nuScenes future semantic segmentation · nuScenes | IoU: 38.63 -> 32.74 | Hurt |
| Replace the ANN baseline with SAD | nuScenes future instance segmentation · nuScenes | PQ: 31.72 -> 20.00 | Hurt |
| Replace the ANN baseline with SAD | nuScenes planning · nuScenes | L2 3s (m): 2.90 -> 3.21 | Hurt |
| Use SAD instead of Freespace | nuScenes planning · nuScenes | Energy (mJ): 344.11 -> 46.92 | Helped |
| Use dense layers instead of self-attention in STM | STM on ImageNet-1K · ImageNet-1K | Accuracy (%): 70.2 -> 72.1 | Helped |
| Use SR for the encoder instead of SA (ablation) | SAD perception encoder · nuScenes perception | Avg. IoU: 16.28 -> 34.06 | Helped |
| Use SA for the decoder instead of SR (ablation) | SAD perception decoder · nuScenes perception | Avg. IoU: 0.11 -> 34.06 | Helped |
| Remove the decoder's timestep connections (ablation) | SAD perception decoder · nuScenes perception | Avg. IoU: 34.06 -> 31.99 | Hurt |
| Replace the MS-ResNet decoder with SEW-ResNet (ablation) | SAD planning model · nuScenes planning ablation | PQ: 68.16 -> 59.28 | Hurt |
| Replace dual-pathway prediction with single-pathway prediction (ablation) | SAD planning model · nuScenes planning ablation | PQ: 68.16 -> 67.55 | Hurt |
Surprising findings
- Replacing self-attention in the spiking token mixer with dense layers improved accuracy, reaching 72.1% ImageNet top-1 with 12M parameters.
- Repeating frames over timesteps in the decoder yields 0.11 avg IoU, while the single-timestep ablation performs only slightly worse than the baseline.
Limitations
- Prediction performance lags behind state-of-the-art ANN methods such as ST-P3.
- Energy numbers are theoretical estimates from SOP/FLOP counts rather than measured on neuromorphic hardware.
- Planning collision rate and L2 are comparable to but not better than ANN baselines; SAD is worse than ST-P3 at 1s L2 (1.53 vs 1.33).
Experimental setup
Datasets: ImageNet-1K nuScenes
Benchmarks: nuScenes
Metrics: IoU PQ SQ RQ L2 error collision rate energy (mJ) ImageNet top-1 accuracy
Baselines: VED VPN PON Lift-Splat IVMP FIERY ST-P3 NMP Freespace Static
Builds on: ST-P3 MS-ResNet SEW-ResNet Spikformer Lift-Splat
Closest papers by meaning
- Adaptive Surrogate Gradients for Sequential Reinforcement Learning in Spiking Neural Networks (NeurIPS 2025 · similarity 0.85)
- Prioritizing Perception-Guided Self-Supervision: A New Paradigm for Causal Modeling in End-to-End Autonomous Driving (NeurIPS 2025 · similarity 0.84)
- A Scalable, Causal, and Energy Efficient Framework for Neural Decoding with Spiking Neural Networks (NeurIPS 2025 · similarity 0.84)
- DriveGazen: Event-Based Driving Status Recognition Using Conventional Camera (AAAI 2025 · similarity 0.84)
- Decision SpikeFormer: Spike-Driven Transformer for Decision Making (CVPR 2025 · similarity 0.84)
- The Dormant Spiking Neuron: A State-Driven Mechanism for Efficient Spiking Neural Networks (NeurIPS 2026 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.