Research map / Video / Video segmentation and tracking
HARDVS: Revisiting Human Activity Recognition with Dynamic Vision Sensors
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces HARDVS, a 300-class, ~107K-sample real event-camera human activity recognition dataset, and ESTF, a dual spatial-temporal Transformer baseline built on ResNet-18 StemNet.
Problem
Human activity recognition with event cameras lacks a large-scale, real-world benchmark; existing event datasets are small, simulated, or saturated.
Approach
Record a large real DAVIS346 event dataset spanning many challenging factors, and propose ESTF, which encodes event frames into spatial and temporal token sequences processed by separate Transformers with a FusionFormer that exchanges information between the two branches.
Key results
- ESTF reaches 51.22 top-1 / 57.53 top-5 on HARDVS, comparable to TimeSformer (50.77/58.70) and Video-SwinTrans (51.91/59.11).
- ESTF achieves 0.832 top-1 on N-Caltech101 and 0.999 top-1 on ASL-DVS, reported as new state of the art.
- Ablation on N-Caltech101 shows the ResNet18 baseline at 72.14 top-1 rising to 83.17 with TF, SF and FusionFormer combined.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use ESTF rather than VMV-GCN | event classification · N-Caltech101 | top-1 accuracy: 0.778 -> 0.832 | Helped |
| use ESTF rather than VMV-GCN | event classification · ASL-DVS | top-1 accuracy: 0.989 -> 0.999 | Helped |
| use ESTF rather than TSM | event recognition · HARDVS | top-1 accuracy: 52.63 -> 51.22 | Hurt |
| add TemporalFormer (TF) (ablation) | ResNet18 · N-Caltech101 | top-1 accuracy: 72.14 -> 81.54 | Helped |
| add SpatialFormer (SF) (ablation) | ResNet18 · N-Caltech101 | top-1 accuracy: 72.14 -> 80.47 | Helped |
| add FusionFormer (ablation) | ESTF with spatial and temporal branches · N-Caltech101 | top-1 accuracy: 82.89 -> 83.17 | Helped |
| add TemporalFormer (TF) (ablation) | ResNet18 · HARDVS | top-1 accuracy: 49.20 -> 49.65 | Helped |
| add SpatialFormer (SF) (ablation) | ResNet18 · HARDVS | top-1 accuracy: 49.20 -> 50.81 | Helped |
| add FusionFormer (ablation) | ESTF with spatial and temporal branches · HARDVS | top-1 accuracy: 51.06 -> 51.22 | Helped |
| use 8 rather than 4 input frames (ablation) | ESTF · N-Caltech101 | mean accuracy: 73.67 -> 77.37 | Helped |
| increase input frames beyond 8 (ablation) | ESTF · N-Caltech101 | mean accuracy: 77.37; 73.49; 75.11; 73.03 | Mixed |
| use 1 rather than 4 Transformer layers (ablation) | ESTF · N-Caltech101 | mean accuracy: 74.64 -> 77.37 | Helped |
Surprising findings
- Strong video models (TimeSformer, Video-SwinTrans, SlowFast) reach only about 50-52% top-1 on HARDVS, showing the dataset is far from saturated.
- On HARDVS, the ESTF gain over the ResNet18 baseline is small (51.22 vs 49.20 top-1), while the gain on N-Caltech101 is large.
Limitations
- Deeper Transformer stacks (4 layers) degrade performance, which the authors attribute to dataset size.
- Only DVS event data is used; RGB frames from the DAVIS346 are not exploited in this work.
- Event streams are converted to 8 stacked event frames, which may discard fine temporal detail.
Experimental setup
Datasets: HARDVS N-Caltech101 ASL-DVS
Benchmarks: HARDVS N-Caltech101 ASL-DVS
Metrics: top-1 accuracy top-5 accuracy mean accuracy
Baselines: C3D R2Plus1D TSM ACTION-Net TAM Video-SwinTrans TimeSformer SlowFast ResNet18 VMV-GCN EV-VGCNN M-LSTM
Builds on: ResNet Vision Transformer TimeSformer
Closest papers by meaning
- DarkShake-DVS: Event-based Human Action Recognition under Low-light and Shaking Camera Conditions (CVPR 2026 · similarity 0.88)
- SMV-EAR: Bring Spatiotemporal Multi-View Representation Learning into Efficient Event-Based Action Recognition (CVPR 2026 · similarity 0.88)
- PASS: Path-selective State Space Model for Event-based Recognition (NeurIPS 2025 · similarity 0.86)
- Scalable Hierarchical Deep Neural Network for Time Series Analysis in Wearable Sensor-based Human Activity Recognition (AAAI 2025 · similarity 0.85)
- MoPFormer: Motion-Primitive Transformer for Wearable-Sensor Activity Recognition (NeurIPS 2025 · similarity 0.85)
- Are Spatial-Temporal Graph Convolution Networks for Human Action Recognition Over-Parameterized? (CVPR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.