Research map / Detection and segmentation / CNN features
Hybrid Spiking Vision Transformer for Object Detection with Event Cameras
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes HsVT, a hybrid ANN-SNN multi-stage vision Transformer (MaxViT spatial blocks plus LSTM/spiking temporal modules) for event-based object detection, and releases a simulated Fall Detection event dataset.
Problem
Event-based object detection needs to capture both spatial and temporal structure in asynchronous event streams efficiently, with low parameter count and energy use.
Approach
Combines MaxViT block/grid self-attention and spiking MLPs for spatial features with LSTM (early blocks) and a spiking temporal feature extraction module (last block) for temporal features, propagating states horizontally across time steps within a RVT-style YOLOX detector.
Key results
- On GEN1, HsVT-B reaches 0.478 mAP_50:95 with 17.2M params, versus RVT at 0.472 with 18.5M params.
- On the Fall dataset, HsVT-tiny reaches 0.491 mAP vs RVT-tiny at 0.487.
- On the Aircraft dataset, HsVT-tiny reaches 0.641 vs RVT-tiny at 0.613.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace RVT with HsVT | RVT · AIR | mAP: 0.613 -> 0.641 | Helped |
| Replace RVT with HsVT | RVT · FALL | mAP: 0.487 -> 0.491; 0.466 -> 0.492; 0.469 -> 0.486 | Mixed |
| Use a 200ms event accumulation interval instead of 40ms or 1000ms (ablation) | HsVT · Fall Detection | mAP: 0.445; 0.487; 0.405 | Helped |
| Add LIF neurons instead of IF neurons in SpikingMLP (ablation) | HsVT · AIR and FALL | mAP: AIR: 0.587 -> 0.630; FALL: 0.441 -> 0.476 | Helped |
| Replace ATan with Sigmoid surrogate gradient (ablation) | HsVT · FALL 1000ms | mAP: 0.476 -> 0.482 | Helped |
| Replace ATan with Sigmoid surrogate gradient (ablation) | HsVT · FALL 200ms and AIR | mAP: FALL 200ms: 0.490 -> 0.473; AIR: 0.630 -> 0.603 | Hurt |
| Use LIFNode instead of IFNode in STFE (ablation) | HsVT · Aircraft Detection | mAP50:95: 0.595 -> 0.640 | Helped |
| Replace STFE with PlainNet + LIFNode (ablation) | HsVT · Aircraft Detection | mAP50:95: 0.640 -> 0.614 | Hurt |
| Replace STFE with FeedBackNet + LIFNode (ablation) | HsVT · Aircraft Detection | mAP50:95: 0.640 -> 0.593 | Hurt |
| Replace STFE with StatefulSynapse + LIFNode (ablation) | HsVT · Aircraft Detection | mAP50:95: 0.640 -> 0.604 | Hurt |
| Replace the block 4 LSTM with STFE (ablation) | HsVT · Aircraft Detection | mAP50:95: 0.595 -> 0.640 | Helped |
| Replace LSTM with STFE in all four blocks (ablation) | HsVT · Aircraft Detection | mAP50:95: 0.595 -> 0.602 | Hurt |
Surprising findings
- Performance did not increase monotonically with model size on the Fall and Aircraft datasets; the base variant underperformed small/tiny, attributed to overfitting from small dataset size.
- The paper's own GEN1 table lists SpikSSD at 40.8 mAP (apparently a units error), and HsVT-B's estimated energy (134.5 mJ) is higher than SFOD and EAS-SNN-M.
Limitations
- Gains over RVT are small on the Fall and Aircraft datasets, and no variance or repeated runs are reported.
- Larger variants do not improve on small datasets, suggesting overfitting or dataset-size limits.
- Estimated energy of the hybrid model is higher than some pure SNN detectors, and the Aircraft dataset is not public.
Experimental setup
Datasets: GEN1 Fall Detection Dataset (simulated from Le2i) Aircraft Detection Dataset
Benchmarks: GEN1 Fall Detection Dataset Aircraft Detection Dataset
Metrics: mAP_50:95 params FLOPs energy (mJ)
Baselines: RVT SpikingViT SpikSSD EAS-SNN SFOD EMS-ResNet34 ERGO-12 STAT
Builds on: RVT MaxViT YOLOX ESIM
Closest papers by meaning
- Efficient Event-Based Object Detection: A Hybrid Neural Network with Spatial and Temporal Attention (CVPR 2025 · similarity 0.91)
- CREST: An Efficient Conjointly-trained Spike-driven Framework for Event-based Object Detection Exploiting Spatiotemporal Dynamics (AAAI 2025 · similarity 0.90)
- SFOD: Spiking Fusion Object Detector (CVPR 2024 · similarity 0.90)
- Object Detection using Event Camera: A MoE Heat Conduction based Detector and A New Benchmark Dataset (CVPR 2025 · similarity 0.89)
- Fully Spiking Neural Networks for Unified Frame-Event Object Tracking (NeurIPS 2025 · similarity 0.88)
- SpikePoint: An Efficient Point-based Spiking Neural Network for Event Cameras Action Recognition (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.