Research map / Video / Video segmentation and tracking
Fully Spiking Neural Networks for Unified Frame-Event Object Tracking
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SpikeFET, the first fully spiking frame-event tracker combining convolutional and Transformer spike blocks, with a Random Patchwork Module to counter padding-induced positional bias and a Spatial-Temporal Regularization loss for template consistency.
Problem
Frame-event object tracking with SNNs is underdeveloped; existing fusion trackers are computationally expensive and SNN trackers are unimodal or hybrid ANN-SNN, and convolutional padding breaks translation invariance in tracking.
Approach
A fully spiking dual-single-dual architecture (ConvFormer spike blocks for local features, TransFormer spike blocks with CSWin spiking self-attention for fusion) plus RPM, which randomly concatenates template and search patches so the target appears at varied positions without removing padding, and STR, which applies an MSE consistency loss between two temporal template features.
Key results
- SpikeFET-Tiny reaches 68.5% AUC and 96.5% PR on FE108, matching HRMonTrack's AUC and 2.7% above SDSTrack with about 39x lower power consumption (18.36 mJ).
- SpikeFET-Base reaches 68.5% AUC on COESOT, 2.7% above MMHT, and 68.7% AUC on FE108.
- SpikeET (event-only) reaches 64.4% AUC on FE108 and 39.4% AUC on VisEvent, outperforming SDTrack by 4.5% and 2.0% respectively.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add RPM to the six-input baseline (ablation) | SpikeFET · FE108 | AUC (%): 32.1 -> 66.8 | Helped |
| add RPM to the six-input baseline (ablation) | SpikeFET · VisEvent | AUC (%): 45.1 -> 55.5 | Helped |
| add STR to RPM (ablation) | SpikeFET · FE108 | AUC (%): 66.8 -> 67.4 | Helped |
| add type encoding and STR to RPM (ablation) | SpikeFET · FE108 | AUC (%): +1.7% | Helped |
| add type encoding and STR to RPM (ablation) | SpikeFET · VisEvent | AUC (%): +1.3% | Helped |
| use four inputs instead of six with RPM (ablation) | SpikeFET · FE108 | AUC (%): 66.1 -> 66.8 | Hurt |
| use frame-event fusion instead of frame-only input (ablation) | SpikeFET-Tiny · FE108 | AUC (%): 51.8 -> 68.5 | Helped |
| use frame-event fusion instead of event-only input (ablation) | SpikeFET-Tiny · FE108 | AUC (%): 64.4 -> 68.5 | Helped |
| concatenate modalities before feature extraction (ablation) | SpikeFET-Tiny · FE108 | AUC (%): 68.5 -> 67.5 | Hurt |
| add modality features instead of concatenating them (ablation) | SpikeFET-Tiny · FE108 | AUC (%): 68.5 -> 68.0 | Hurt |
| remove modality encodings (ablation) | SpikeFET-Tiny · FE108 | AUC (%): 68.5 -> 68.0 | Hurt |
| remove dual-head tracking (ablation) | SpikeFET-Tiny · FE108 | AUC (%): 68.5 -> 67.0 | Hurt |
Surprising findings
- SpikeFET-Base underperforms ViPT and SDSTrack on VisEvent (59.0% vs 59.2-59.7% AUC for ViPT and SDSTrack), which the authors attribute to limited tracking pre-training for SNNs.
- Concatenating modality images before feature extraction degrades performance relative to independent modality branches.
Limitations
- Base model shows overfitting tendency and underperforms on VisEvent due to limited tracking-specific pre-training
- Energy results are theoretical estimates rather than measured on neuromorphic hardware
- Deployment on edge or neuromorphic chips not evaluated
Experimental setup
Datasets: FE108 VisEvent COESOT COCO LaSOT TrackingNet GOT-10K ImageNet-1K
Benchmarks: FE108 VisEvent COESOT
Metrics: AUC PR power consumption (mJ) parameters (M)
Baselines: ViPT SDSTrack MMHT CEUTrack OSTrack HRMonTrack SDTrack SNNTrack
Builds on: OSTrack CEUTrack SDTrack CSWin Transformer Spike Firing Approximation
Closest papers by meaning
- Leveraging Asynchronous Spiking Neural Networks for Ultra Efficient Event-Based Visual Processing (AAAI 2025 · similarity 0.89)
- Spiking Neural Network as Adaptive Event Stream Slicer (NeurIPS 2024 · similarity 0.88)
- CREST: An Efficient Conjointly-trained Spike-driven Framework for Event-based Object Detection Exploiting Spatiotemporal Dynamics (AAAI 2025 · similarity 0.88)
- SFOD: Spiking Fusion Object Detector (CVPR 2024 · similarity 0.88)
- SpikePoint: An Efficient Point-based Spiking Neural Network for Event Cameras Action Recognition (ICLR 2024 · similarity 0.88)
- Hybrid Spiking Vision Transformer for Object Detection with Event Cameras (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.