Research map / Architectures and efficiency / RNNs and normalization
State Space Models for Event Cameras
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces SSM-ViT, a hybrid of state-space layers (S4D/S5) and vision transformer blocks for event-camera object detection, using learnable timescales to generalize across inference frequencies and two anti-aliasing strategies (output bandlimiting and H2 norm).
Problem
Event-camera detectors trained on event representations at one temporal window size (frequency) degrade sharply when deployed at higher inference frequencies, and recurrent training is slow.
Approach
Replace RNN temporal aggregation with linear state-space layers whose learnable timescale can be rescaled at inference to match new frequencies, combined with frequency-selective output masking and an H2-norm loss to suppress aliasing.
Key results
- SSM-based models have an average mAP drop of 3.31 between training and higher test frequencies, versus 21.25 for RVT and 24.53 for GET.
- S5-ViT-B reaches 47.4 mAP on Gen1 and 47.2 on 1 Mpx at 20 Hz with 8.16 ms runtime on Gen1, competitive with prior SotA.
- SSM-ViT trains up to 33% faster than recurrent vision transformer approaches while keeping comparable accuracy.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace recurrent layers with SSM temporal aggregation | Event-camera object detection models · Gen1 and 1 Mpx, evaluation at higher frequencies | mAP performance drop: 3.31 mAP vs 21.25 and 24.53 mAP | Helped |
| Use S5 instead of RVT | Event-camera object detection · Gen1 and 1 Mpx validation, 20–200 Hz | performance drop: RVT 27.62, 29.42, 16.00; S5 H2 norm 4.23, 4.72, 3.11 | Helped |
| Apply frequency-selective masking with α=0.5 (ablation) | S4D and S5 · Gen1 val | mAP: S4D-legS: 46.93 -> 47.33; S5-legS: 48.33 -> 48.48 | Mixed |
| Replace legS initialization with inv initialization (ablation) | S4D and S5 · Gen1 val | mAP at α=0.5: 47.33 -> 46.23; 48.48 -> 47.43 | Hurt |
| Replace legS initialization with lin initialization (ablation) | S4D and S5 · Gen1 val | mAP at α=0.5: 47.33 -> 46.02; 48.48 -> 46.40 | Hurt |
| Remove SSM recurrence from all stages (ablation) | SSM-ViT · Gen1 val | mAP: 33.90 vs 47.33 | Hurt |
| Add SSM recurrence beginning at stage 4 (ablation) | SSM-ViT · Gen1 val | mAP: 33.90 -> 39.99 | Helped |
| Add SSM recurrence in stages 1–4 rather than stages 1–3 (ablation) | SSM-ViT · Gen1 val | mAP: S4D: 47.02 -> 47.33; S5: 48.41 -> 48.48 | Mixed |
| Use H2 norm rather than α=0.5 masking (ablation) | S4D and S5 · Gen1 and 1 Mpx validation, 20–200 Hz | performance drop: S4D Gen1: 5.31 vs 5.61; S4D 1Mpx: 4.66 vs 5.77; S5 Gen1: 4.23 vs 4.30; S5 1Mpx: 3.11 vs 2.82 | Mixed |
| Use SSM-based model instead of RNN/Transformer recurrent approaches | Event-camera object detection · Training | training speed: 33% faster | Helped |
Surprising findings
- A linear SSM temporal module reaches comparable detection accuracy to the non-linear LSTM used in RVT, suggesting the LSTM complexity may not be needed.
Limitations
- Does not outperform top methods ERGO-12 and GET-T in absolute mAP on Gen1/1 Mpx.
- Aliasing at high frequencies still causes some performance drop, mitigated but not eliminated by the proposed strategies.
Experimental setup
Datasets: Gen1 Automotive Detection 1 Mpx detection dataset DSEC
Benchmarks: Gen1 1 Mpx DSEC
Metrics: mAP runtime (ms) performance drop across frequencies
Baselines: RVT-B GET-T Swin-T v2 Nested-T ERGO-12 RED S4D-ViT-B
Builds on: S4 S4D S5 RVT YOLOX
Closest papers by meaning
- PASS: Path-selective State Space Model for Event-based Recognition (NeurIPS 2025 · similarity 0.88)
- From Layers to States: A State Space Model Perspective to Deep Neural Network Layer Dynamics (ICLR 2025 · similarity 0.86)
- EventZoom: A Progressive Approach to Event-Based Data Augmentation for Enhanced Neuromorphic Vision (AAAI 2025 · similarity 0.85)
- Leveraging Asynchronous Spiking Neural Networks for Ultra Efficient Event-Based Visual Processing (AAAI 2025 · similarity 0.85)
- A Simple and Effective Point-based Network for Event Camera 6-DOFs Pose Relocalization (CVPR 2024 · similarity 0.85)
- E2HQV: High-Quality Video Generation from Event Camera via Theory-Inspired Model-Aided Deep Learning (AAAI 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.