Research map / Architectures and efficiency / Attention and transformers
ALERT-Transformer: Bridging Asynchronous and Synchronous Machine Learning for Real-Time Event-based Spatio-Temporal Data
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ALERT-Transformer, a hybrid pipeline where a PointNet-based event embedding with leaky max-pooling token updates lets a ViT-style transformer classify event streams asynchronously at any sampling rate.
Problem
Event-based sensors produce ultra-sparse asynchronous streams, but dense ML models need synchronous tensor inputs; existing conversions either wait to accumulate events (high latency) or require spiking networks or specialized async architectures.
Approach
A PointNet-based patch embedding (LERT/ALERT) with sinusoidal time encoding, channel-wise max-pooling updated per incoming event, and an Old Maximum Value Decay leak on tokens, so a standard transformer can read always-up-to-date token features at any sampling rate.
Key results
- ALERT-Transformer RM reaches 84.6% sample accuracy on DVS128Gesture with 9.6 ms inference time, versus 79.8% for the recurrent Event Transformer under per-24ms prediction.
- LMM (0.57M params, 0.0074 MFLOPs/event) reaches 85.6% accuracy on N-Cars, a 75% complexity reduction versus the prior best asynchronous model at a 5.11% accuracy drop.
- Synchronous LERT-Transformer RM achieves 96.2% file voting accuracy on DVS128Gesture; ALERT asynchronous inference gives 94.1%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ALERT with time encoding and decay instead of synchronous LERT | Reference Model · DVS128Gesture | FVA: 96.2% -> 94.1% | Hurt |
| Use ALERT with time encoding and decay instead of synchronous LERT | Reference Model · DVS128Gesture | SA: 88.6% -> 84.6% | Hurt |
| Use ALERT instead of LERT | Low-Memory Model · DVS128Gesture | FVA: 92.0% -> 89.2% | Hurt |
| Use sinusoidal time encoding (TELERT) instead of ordinary LERT | Reference Model · DVS128Gesture | FVA: 0.4% | No clear effect |
| Remove OMVD by setting λ = 0 (ablation) | ALERT-Transformer RM · DVS128Gesture | SA: 25.9% | Hurt |
| Remove positional embeddings (ablation) | LERT-Transformer RM · DVS128Gesture | SA: 88.6% -> 80.0% | Hurt |
| Remove positional embeddings (ablation) | LERT-Transformer RM · DVS128Gesture | FVA: 96.2% -> 95.1% | Hurt |
| Vary input sequence length from 8192 events (ablation) | ALERT-Transformer RM · DVS128Gesture | relative accuracy: 0.963% to 0.997% | Mixed |
| Increase patch size from [8,8] (ablation) | ALERT-Transformer RM · DVS128Gesture | relative accuracy: 1.000% -> 0.973% (12,12); 1.000% -> 0.999% (32,32) | Mixed |
| Vary voxel activation rate from 0.75 (ablation) | ALERT-Transformer RM · DVS128Gesture | relative accuracy: 1.000% -> 0.970% to 0.993% | Mixed |
| Use CTIM with Δt = 180ms instead of CCIM with Ne=8192 (ablation) | LERT-Transformer · DVS128Gesture | SA: 88.5% SA and 96.5% FVA | Mixed |
Surprising findings
- Without the decay (lambda = 0), ALERT sample accuracy collapses to 25.9%, showing old-information forgetting is essential.
Limitations
- Per-event asynchronous feature generation is too slow in software, so asynchronous inference is simulated by on-demand token updates rather than run truly event-by-event.
- Asynchronous (SA) accuracy drops notably versus file-level voting (e.g., 84.6% vs 96.2% for RM; 72.9% vs 92.0% for LMM), and the LMM struggles on gesture recognition.
- Evaluated only on classification tasks with a transformer head; generalization to other tasks and architectures is untested.
Experimental setup
Datasets: DVS128Gesture N-Cars
Benchmarks: DVS128Gesture N-Cars
Metrics: File Voting Accuracy Sample Accuracy N-Voting Accuracy Time to Accuracy FLOPs per event parameters inference time
Baselines: 3D-CNN GET PointNet++ EventTransAct Event Transformer CNN + LSTM HOTS HATS Async. Sparse CNNs YOLE EST
Builds on: PointNet PointBERT EventNet Vision Transformer Asynchronous Sparse Convolutional Networks
Closest papers by meaning
- EGSST: Event-based Graph Spatiotemporal Sensitive Transformer for Object Detection (NeurIPS 2024 · similarity 0.85)
- SMamba: Sparse Mamba for Event-based Object Detection (AAAI 2025 · similarity 0.85)
- Scene Adaptive Sparse Transformer for Event-based Object Detection (CVPR 2024 · similarity 0.83)
- PASS: Path-selective State Space Model for Event-based Recognition (NeurIPS 2025 · similarity 0.83)
- PLEIADES: Building Temporal Kernels with Orthogonal Polynomials (NeurIPS 2025 · similarity 0.83)
- HARDVS: Revisiting Human Activity Recognition with Dynamic Vision Sensors (AAAI 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.