Research map / Video / Video segmentation and tracking

SpikeVideoFormer: An Efficient Spike-Driven Video Transformer with Hamming Attention and $\mathcal{O}(T)$ Complexity

ICML 2025 · Video › Video segmentation and tracking · approach: Self-supervised video representation learning

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes SpikeVideoFormer, a directly trained spike-driven video Transformer with Hamming-similarity attention (SDHA) and joint space-time attention giving O(T) temporal complexity, evaluated on Kinetics-400 classification, MMHPSD pose tracking, and Cityscapes/VSPW video segmentation.

Problem

Existing spike-driven Transformers target single images, use dot-product attention scores that fail to capture similarity between binary spike vectors, and do not explore efficient spike-driven space-time attention for video tasks.

Approach

Replace the dot-product attention score in spike-driven attention with normalized Hamming similarity (theoretically justified via a binary-embedding JL lemma), rewritten as a ±1 dot product so it can be computed with linear complexity by reordering matrix multiplications, and use joint space-time attention over the TN tokens.

Key results

• Kinetics-400 top-1 accuracy of 79.8% (SNN), surpassing Meta-SpikeFormer's 75.5% by 4.3 points and trailing Swin-B (80.6%) by 0.8 points.
• MMHPSD pose tracking (T=32, video) PA-MPJPE 47.5 vs 57.1 for the best prior SNN; Cityscapes VSS mIoU 73.1 vs 65.9 for Meta-SpikeFormer.
• Replacing Hamming attention with dot-product drops Cityscapes VSS mIoU from 73.1 to 65.9 and raises pose PA-MPJPE from 39.8 to 45.7.

Measured effects reported in the paper

Surprising findings

• Dot-product attention masks spike outputs so the reduced power from the ablation is attributed to lower spiking rate, while accuracy drops significantly.
• The directly trained SNN reaches 79.9% top-1 on ImageNet-1K, reported as surpassing ViT (79.7%).

Limitations

• Video segmentation uses a deliberately simple head, so it trails complex ANN models like CFFM on VSPW (37.9 vs 49.3 mIoU).
• On Kinetics-400 the model still trails the ANN Swin-B by 0.8% top-1 and pose tracking remains behind GLoT at T=32.
• Hamming-approximation proposition relies on a large D and the hyperparameter scale s is set empirically per layer.

Experimental setup

Datasets: ImageNet-1K Kinetics-400 MMHPSD Cityscapes VSPW ImageNet-C

Benchmarks: Kinetics-400 MMHPSD Cityscapes VSPW ImageNet-1K ImageNet-C

Metrics: top-1 accuracy top-5 accuracy MPJPE PEL-MPJPE PA-MPJPE mIoU power (mJ) parameter count top-1 accuracy on ImageNet

Baselines: I3D-101 ViViT Swin-B Meta-SpikeFormer VIBE GLoT SEW-Res MA-SNN SpikFormer FCN PSPNet SegFormer

Builds on: Spike-driven Self-Attention (SDSA) Meta-SpikeFormer Spikformer Johnson-Lindenstrauss lemma for binary embedding Leaky Integrate-and-Fire neuron

Closest papers by meaning

• Spiking Vision Transformer with Saccadic Attention (ICLR 2025 · similarity 0.90)
• SpikedAttention: Training-Free and Fully Spike-Driven Transformer-to-SNN Conversion with Winner-Oriented Spike Shift for Softmax Operation (NeurIPS 2024 · similarity 0.89)
• Spike2Former: Efficient Spiking Transformer for High-performance Image Segmentation (AAAI 2025 · similarity 0.88)
• Finding Visual Saliency in Continuous Spike Stream (AAAI 2024 · similarity 0.88)
• Inference-Scale Complexity in ANN-SNN Conversion for High-Performance and Low-Power Applications (CVPR 2025 · similarity 0.88)
• Spike-driven Transformer V2: Meta Spiking Neural Network Architecture Inspiring the Design of Next-generation Neuromorphic Chips (ICLR 2024 · similarity 0.87)