Research map / Video / Action recognition
ExACT: Language-guided Conceptual Reasoning and Uncertainty Estimation for Event-based Action Recognition and More
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ExACT, a language-guided event-based action recognition framework with adaptive fine-grained event frames, conceptual-reasoning fusion and distributional uncertainty estimation, and releases the SeAct caption-labeled event dataset.
Problem
Event-based action recognition is hard because actions have long durations with complex, ambiguous semantics, and fixed-stack event frame representations produce redundant, overlapped frames.
Approach
Uses a pre-trained event-text model (ECLIP) with an adaptive event-frame sampler (binary-search on event-count difference maps) and a CRUE module that fuses frames by text-guided conceptual reasoning weights and models semantic uncertainty via Gaussian distributional embeddings trained with contrastive, smooth L1 and variance regularization losses.
Key results
- ExACT reaches 94.83% Top-1 on PAF (+2.23% over prior SOTA) and 90.10% on HARDVS (+37.47%).
- On DVS128 Gesture ExACT reports 98.86%; on the new SeAct dataset it reaches 67.24% Top-1 with caption labels.
- Ablation: CR fusion adds +0.78% and CR with uncertainty estimation adds +1.97% Top-1 over the contrastive baseline on PAF.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ExACT instead of EV-ACT | Event-based action recognition · PAF | Top-1 accuracy: 92.60 -> 94.83 | Helped |
| Use ExACT instead of TSM | Event-based action recognition · HARDVS | Top-1 accuracy: 52.63 -> 90.10 | Helped |
| Use ExACT instead of EventTransAct | Event-based action recognition · DVS128 Gesture | Top-1 accuracy: 97.92 -> 98.86 | Helped |
| Train ExACT with caption-level rather than category-level labels (ablation) | ExACT · SeAct | Top-1 accuracy: 66.07 -> 67.24 | Helped |
| Use adaptive event sampling instead of fixed-count or fixed-time sampling (ablation) | Event-frame representation on PAF · PAF | Top-1 accuracy: 92.79 -> 94.83 | Helped |
| Add conceptual reasoning-based fusion (ablation) | Contrastive ExACT on PAF · PAF | Top-1 accuracy: 92.86 -> 93.64 | Helped |
| Add uncertainty estimation to conceptual reasoning (ablation) | Contrastive ExACT with CR on PAF · PAF | Top-1 accuracy: 93.64 -> 94.83 | Helped |
| Replace sum fusion with conceptual reasoning (ablation) | Event-frame fusion on PAF · PAF | Top-1 accuracy: 89.14 -> 94.83 | Helped |
| Replace mean pooling with conceptual reasoning (ablation) | Event-frame fusion on PAF · PAF | Top-1 accuracy: 92.52 -> 94.83 | Helped |
| Add Lreg to contrastive training (ablation) | ExACT training objectives on PAF · PAF | Top-1 accuracy: 92.86 -> 93.96 | Helped |
| Add LsmoothL1 to contrastive training (ablation) | ExACT training objectives on PAF · PAF | Top-1 accuracy: 92.86 -> 94.41 | Helped |
| Increase sampled discrete embeddings from N=2 to N=6 (ablation) | CRUE module on PAF · PAF | Top-1 accuracy: 92.43 -> 94.83 | Helped |
Surprising findings
- Mean pooling of frames already outperforms the sum fusion by a large margin, and CR adds a further gain over mean pooling on PAF.
Limitations
- SeAct accuracy (67.24%) remains modest, indicating the dataset is challenging.
- Distributional sampling increases training complexity, especially given event sparsity.
- AFE hyperparameters (Nmin, threshold) vary per dataset and must be tuned.
Experimental setup
Datasets: PAF HARDVS DVS128 Gesture SeAct
Benchmarks: PAF HARDVS DVS128 Gesture SeAct
Metrics: Top-1 accuracy Top-5 accuracy retrieval similarity (qualitative)
Baselines: EV-ACT MST Swin-T (BN) Motion SNN TSM Video-SwinTrans TBR EventTransAct EvT ESTF TAM
Builds on: ECLIP contrastive learning distributional representation
Closest papers by meaning
- Talk2Event: Grounded Understanding of Dynamic Scenes from Event Cameras (NeurIPS 2025 · similarity 0.87)
- PASS: Path-selective State Space Model for Event-based Recognition (NeurIPS 2025 · similarity 0.86)
- Black Swan: Abductive and Defeasible Video Reasoning in Unpredictable Events (CVPR 2025 · similarity 0.86)
- Storyboard-guided Alignment for Fine-grained Video Action Recognition (NeurIPS 2025 · similarity 0.85)
- SeFAR: Semi-supervised Fine-grained Action Recognition with Temporal Perturbation and Learning Stabilization (AAAI 2025 · similarity 0.85)
- Event-Grounded Sparse Autoencoders for Vision-Language-Action Policies (NeurIPS 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.