Research map / Detection and segmentation / CNN features
Scene Adaptive Sparse Transformer for Event-based Object Detection
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SAST, a hierarchical window-based Transformer for event-based object detection that co-sparsifies windows and tokens with learned, scene-adaptive scoring and masked sparse window self-attention.
Problem
Transformers for event-based object detection are computationally expensive, eroding the low-power advantage of event cameras; existing image sparse Transformers either sparsify tokens at fine granularity with costly global attention or prune windows with a fixed ratio, lacking scene adaptivity.
Approach
Scores each window and token with a learned scoring module modulated by event sparsity, selects windows and tokens via competition-based normalization and thresholds, and runs Masked Sparse Window Self-Attention on the unequal-sized selected token sets so computation adapts to scene complexity.
Key results
- SAST reaches 48.3% mAP on 1Mpx with 1.8G A-FLOPs, about 28% of RVT's A-FLOPs in the table's FLOPs comparison (5.6G total backbone FLOPs vs 10.3G for RVT).
- SAST-CB reaches 48.7% mAP on 1Mpx and 48.2% on Gen1, surpassing RVT's 47.4% and 47.2% baselines.
- On Gen1, SAST achieves 47.9% mAP with 2.1G FLOPs, about 64% fewer FLOPs than RVT's 3.5G.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use scene-adaptive window-token co-sparsification | RVT-B / MaxViT · 1Mpx | mAP (%): 47.4 -> 48.3 | Helped |
| Use scene-adaptive window-token co-sparsification | RVT-B / MaxViT · 1Mpx | A-FLOPs (G): 6.5 -> 1.8 | Helped |
| Add context broadcasting to SAST (ablation) | SAST · 1Mpx | mAP (%): 48.3 -> 48.7 | Helped |
| Add context broadcasting to SAST (ablation) | SAST · Gen1 | mAP (%): 47.9 -> 48.2 | Helped |
| Replace the SAST scoring module with L2 activation scoring (ablation) | SAST · 1Mpx | mAP (%): 48.3 -> 45.1 | Hurt |
| Replace the SAST scoring module with attention-mask scoring (ablation) | SAST · 1Mpx | mAP (%): 48.3 -> 45.8 | Hurt |
| Replace the SAST scoring module with head scores (ablation) | SAST · 1Mpx | mAP (%): 48.3 -> 46.8 | Hurt |
| Replace the SAST scoring module with head importance (ablation) | SAST · 1Mpx | mAP (%): 48.3 -> 46.5 | Hurt |
| Add token selection alongside window selection (ablation) | SAST · 1Mpx | mAP (%): 46.3 -> 48.3 | Helped |
| Add window selection alongside token selection (ablation) | SAST · 1Mpx | mAP (%): 47.7 -> 48.3 | Helped |
| Use masked sparse window self-attention instead of standard self-attention (ablation) | SAST · 1Mpx | mAP (%): 40.7 -> 48.3 | Helped |
| Use masked sparse window self-attention instead of sparse self-attention (ablation) | SAST · 1Mpx | mAP (%): 44.2 -> 48.3 | Helped |
Surprising findings
- Ablation shows that selecting both windows and tokens gives higher mAP (48.3%) than either window-only (46.3%) or token-only (47.7%) selection, despite being the sparsest option.
Limitations
- Measured runtime is higher than the RVT baseline (19.7 ms vs 16.0 ms), so FLOP savings do not translate to latency gains.
- Performance on 1Mpx stays slightly below AEC, which uses a heavier representation and augmentation.
- Scene adaptivity is evaluated qualitatively and on a small set of visualized scenes.
Experimental setup
Datasets: 1Mpx Gen1
Benchmarks: 1Mpx Gen1
Metrics: COCO mAP FLOPs A-FLOPs runtime params
Baselines: RVT GET AEC ASTMNet RED SViT AViT SparseTT YOLOv3 DVS SAM
Builds on: RVT SparseViT Swin Transformer Sparse DETR SparseTT
Closest papers by meaning
- SMamba: Sparse Mamba for Event-based Object Detection (AAAI 2025 · similarity 0.88)
- EGSST: Event-based Graph Spatiotemporal Sensitive Transformer for Object Detection (NeurIPS 2024 · similarity 0.87)
- Motion-adaptive Transformer for Event-based Image Deblurring (AAAI 2025 · similarity 0.86)
- FlexEvent: Towards Flexible Event-Frame Object Detection at Varying Operational Frequencies (NeurIPS 2025 · similarity 0.86)
- Context Enhanced Transformer for Single Image Object Detection in Video Data (AAAI 2024 · similarity 0.85)
- LEOD: Label-Efficient Object Detection for Event Cameras (CVPR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.