Research map / Label-efficient and robust learning / Recognition and long-tail
Action-slot: Visual Action-centric Representations for Multi-label Atomic Activity Recognition in Traffic Scenes
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Action-slot, an action-centric slot attention framework for multi-label atomic activity recognition in traffic scenes, plus the synthetic CARLA-based TACO dataset to balance activity classes.
Problem
Multi-label atomic activity recognition of road users in traffic scenes, where existing video action recognizers and object-aware models struggle to decompose multiple concurrent activities, and real-world datasets (OATS) have imbalanced class distributions.
Approach
Allocate one slot per atomic activity class with a parallel spatial-temporal slot update, add a background slot supervised by a background mask, and regularize slots of absent classes to attend nowhere, avoiding object proposals or perception guidance.
Key results
- Action-slot reaches 28.6 mAP on OATS with ImageNet ResNet50 and 48.2 mAP with Kinetics-400 X3D, outperforming OATS (26.7) and CSN (44.8).
- On the TACO dataset Action-slot achieves 54.4 mAP with X3D, versus 44.0 for CSN and 38.3 for X3D.
- Pretraining on TACO improves OATS mAP for Action-slot from 48.2 to 59.5 and nuScenes from 23.6 to 32.3.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Action-slot instead of the strongest listed ImageNet-pretrained baseline | OATS recognition with ResNet50/ImageNet · OATS | mAP: 26.7 -> 28.6 | Helped |
| Use Action-slot instead of the strongest listed Kinetics-400-pretrained baseline | OATS recognition with X3D/ResNet backbones · OATS | mAP: 44.8 -> 48.2 | Helped |
| Use Action-slot instead of CSN | TACO recognition · TACO | mAP: 44.0 -> 54.4 | Helped |
| Pretrain representations on TACO before finetuning | Action-slot · OATS | mAP: 48.2 -> 59.5 (+11.3) | Helped |
| Pretrain representations on TACO before finetuning | Action-slot · nuScenes | mAP: 23.6 -> 32.3 (+8.7) | Helped |
| Add allocated slots (ablation) | Action-slot · OATS | mAP: 10.8 -> 42.7 | Helped |
| Replace recurrent updates with parallel slot updates (ablation) | Action-slot with allocated slots · OATS | mAP: 24.8 -> 42.7 | Helped |
| Add an unsupervised background slot (ablation) | Action-slot with allocated slots and parallel updates · OATS | mAP: 42.7 -> 40.8 | Hurt |
| Add background-mask supervision Lbg (ablation) | Action-slot with allocated slots and parallel updates · OATS | mAP: 42.7 -> 43.6 | Helped |
| Add negative-class attention regularization Lneg (ablation) | Action-slot with allocated slots and parallel updates · OATS | mAP: 42.7 -> 43.0 | Helped |
| Combine allocated slots, parallel updates, background-mask supervision, and negative-class regularization (ablation) | Action-slot · OATS | mAP: 42.7 -> 48.2 | Helped |
Surprising findings
- Adding a non-allocated background slot without mask supervision slightly harms performance (40.8 vs 42.7 mAP).
- Object-level mask guidance with ground-truth instance masks performs worse than the proposed background-and-negative attention guidance on TACO.
Limitations
- Struggles when two atomic activities occlude each other, causing action slots to confuse attention.
- TACO is synthetic (CARLA), so real-world domain gap is only partially addressed by pretraining transfer.
Experimental setup
Datasets: OATS TACO nuScenes Kinetics-400
Benchmarks: OATS TACO nuScenes
Metrics: mAP
Baselines: I3D X3D CSN SlowFast MViT VideoMAE ORN ARG OATS SAVi MO Slot-VPS
Builds on: slot attention SAVi Multi-Object Representation Learning (MO) OATS X3D
Closest papers by meaning
- AVA: A Video Dataset of Spatio-Temporally Localized Atomic Visual Actions (CVPR 2018 · similarity 0.88)
- VA-AR: Learning Velocity-Aware Action Representations with Mixture of Window Attention (AAAI 2025 · similarity 0.84)
- HARDVS: Revisiting Human Activity Recognition with Dynamic Vision Sensors (AAAI 2024 · similarity 0.83)
- SeFAR: Semi-supervised Fine-grained Action Recognition with Temporal Perturbation and Learning Stabilization (AAAI 2025 · similarity 0.83)
- Adaptive Slot Attention: Object Discovery with Dynamic Slot Number (CVPR 2024 · similarity 0.83)
- Dual DETRs for Multi-Label Temporal Action Detection (CVPR 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.