Research map / Video / Video segmentation and tracking
PASS: Path-selective State Space Model for Event-based Recognition
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PASS, a state-space-model (VideoMamba-based) framework for event-camera recognition with a path-selective frame selection module (PEAS) and a multi-faceted selection guiding loss (MSG), targeting long event streams and generalization across inference frequencies.
Problem
Event-based object/action recognition methods handle only a narrow range of event lengths and generalize poorly when inference sampling frequency differs from training frequency.
Approach
Use a differentiable Gumbel-softmax frame selection (PEAS) to map variable-length event frame sequences to fixed-size selected frames, scanned bidirectionally by a Mamba-based SSM, plus an information-entropy/mutual-information loss (MSG) that reduces selection randomness and redundancy.
Key results
- PASS-M-K(2) reaches 94.60% on N-Caltech101 (+2.25%) and 61.32% on N-ImageNet (+3.43%) over prior best.
- PASS-M-K(8) reaches 98.28% on PAF (+3.45%) and PASS-S-K(16) reaches 98.41% on HARDVS (+8.31%) over ExACT.
- Trained at 60 Hz on PAF and evaluated at 100 Hz, PASS drops at most 8.62% versus -20.69% for the event-count baseline.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use PASS instead of ExACT | Event recognition · PAF | Top-1 accuracy (%): +3.45% | Helped |
| use PASS instead of ExACT | Event recognition · SeAct | Top-1 accuracy (%): +0.38% | Helped |
| use PASS instead of ExACT | Event recognition · HARDVS | Top-1 accuracy (%): +8.31% | Helped |
| use PASS instead of EventDance | Event recognition · N-Caltech101 | Top-1 accuracy (%): +2.25% | Helped |
| use PASS instead of MEM | Event recognition · N-Imagenet | Top-1 accuracy (%): +3.43% | Helped |
| use PASS-M-K(32) instead of S5-ViT-B-K(32) | ArDVS100 recognition · ArDVS100 | Top-1 accuracy (%): 97.35% vs. 93.39% | Helped |
| use PASS-T-K(32) instead of S5-ViT-B-K(32) | TemArDVS100 recognition · TemArDVS100 | Top-1 accuracy (%): 89.00% vs. 79.62% | Helped |
| use PASS trained at 60 Hz, evaluated at 100 Hz, instead of the cited baseline | Inference-frequency generalization · PAF | accuracy drop (%): -8.62% vs. -20.69% | Helped |
| use PEAS instead of random frame sampling (ablation) | PASS-T · PAF | Top-1 accuracy (%): 92.98% -> 93.33% (+0.35%) | Helped |
| use PEAS instead of random frame sampling (ablation) | PASS-T · ArDVS100 | Top-1 accuracy (%): 92.23% -> 92.84% (+0.61%) | Helped |
| add MSG loss to PEAS (ablation) | PASS-T · PAF | Top-1 accuracy (%): 93.33% -> 94.83% (+1.85% over random-sampling baseline) | Helped |
| add MSG loss to PEAS (ablation) | PASS-T · ArDVS100 | Top-1 accuracy (%): 92.84% -> 93.85% (+1.62% over random-sampling baseline) | Helped |
Surprising findings
- Larger VideoMamba variants tend to overfit, yielding suboptimal performance.
- The PASS-T-K(16) variant underperforms S5-ViT on ArDVS100 and TemArDVS100 despite lower parameter count, with the gap reversed at K(32).
Limitations
- Larger VideoMamba models overfit and perform worse.
- Direct comparison with S5-ViT (detection-oriented) required substantial re-implementation and replacement of backbone and head.
Experimental setup
Datasets: PAF SeAct HARDVS N-Caltech101 N-ImageNet ArDVS100 TemArDVS100 Real-ArDVS10
Benchmarks: N-Caltech101 N-ImageNet PAF SeAct HARDVS ArDVS100 TemArDVS100 Real-ArDVS10
Metrics: Top-1 accuracy Top-5 accuracy accuracy drop across inference frequencies
Baselines: S5-ViT-B-K ExACT EventDance MEM EST EV-ACT EventTransAct EvT TTPIONT DiST
Builds on: VideoMamba Mamba Gumbel softmax S5-ViT
Closest papers by meaning
- State Space Models for Event Cameras (CVPR 2024 · similarity 0.88)
- EventZoom: A Progressive Approach to Event-Based Data Augmentation for Enhanced Neuromorphic Vision (AAAI 2025 · similarity 0.87)
- FlexEvent: Towards Flexible Event-Frame Object Detection at Varying Operational Frequencies (NeurIPS 2025 · similarity 0.87)
- Event Stream-based Visual Object Tracking: A High-Resolution Benchmark Dataset and A Novel Baseline (CVPR 2024 · similarity 0.87)
- HARDVS: Revisiting Human Activity Recognition with Dynamic Vision Sensors (AAAI 2024 · similarity 0.86)
- ExACT: Language-guided Conceptual Reasoning and Uncertainty Estimation for Event-based Action Recognition and More (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.