Research map / Video / Action recognition
Adapting Short-Term Transformers for Action Detection in Untrimmed Videos
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Adapts a pre-trained plain short-term ViT (VideoMAE) into a unified long-form video transformer for end-to-end temporal action detection, using inner-backbone cross-snippet propagation blocks and post-backbone temporal transformer layers.
Problem
Pre-trained short-term video ViTs are typically used as frozen per-snippet feature extractors for temporal action detection, which ignores inter-snippet relations in untrimmed videos and makes end-to-end training memory-hungry.
Approach
Insert cross-snippet propagation modules (zero-initialized 3D bottleneck convolution "local" blocks or temporal-only self-attention "global" blocks with clip-level positional encoding) evenly inside a plain ViT backbone so snippets exchange information during fine-tuning, plus temporal transformer encoder layers after the backbone, yielding an end-to-end TAD detector ViT-TAD.
Key results
- ViT-TAD with ViT-B (VideoMAE) reaches 69.5 average mAP on THUMOS14, outperforming all end-to-end TAD methods except VideoMAE V2 (ViT-G).
- ViT-TAD with ViT-B achieves 37.40 average mAP on ActivityNet-1.3, state of the art among end-to-end methods.
- ViT-TAD with ViT-B achieves 17.20 average mAP on FineAction, only slightly below VideoMAE V2 (18.24).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add 4 global inner-backbone propagation blocks (ablation) | ViT-TAD · THUMOS14 validation | average mAP: 59.4 -> 62.3 | Helped |
| increase global propagation blocks from 4 to 12 (ablation) | ViT-TAD · THUMOS14 validation | average mAP: 62.3 -> 60.3 | Hurt |
| increase local propagation blocks from 4 to 12 (ablation) | ViT-TAD · THUMOS14 validation | average mAP: 60.8 -> 62.0 | Helped |
| add clip-level temporal positional encoding (ablation) | ViT-TAD global propagation · THUMOS14 validation | average mAP: 61.3 -> 62.3 | Helped |
| replace temporal-only 1D propagation with temporal-and-spatial 3D propagation (ablation) | ViT-TAD · THUMOS14 validation | average mAP: 62.3 -> 62.2 | No clear effect |
| replace evenly placed global blocks with blocks in the first 4 backbone blocks (ablation) | ViT-TAD · THUMOS14 validation | average mAP: 62.3 -> 60.1 | Hurt |
| add post-backbone propagation to 4 global inner-backbone blocks (ablation) | ViT-TAD · THUMOS14 validation | average mAP: 62.3 -> 63.0 | Helped |
| add post-backbone propagation to 12 local inner-backbone blocks (ablation) | ViT-TAD · THUMOS14 validation | average mAP: 62.0 -> 61.5 | Hurt |
| add inner-backbone and post-backbone propagation modules | TALLFormer · THUMOS14 | average mAP: 63.1% to 66.1% | Helped |
| use ViT-TAD with ViT-B instead of VideoMAE V2 with ViT-G | THUMOS14 detectors · THUMOS14 test | average mAP: 69.6 -> 69.5 | Hurt |
| use ViT-TAD with ViT-B instead of VideoMAE V2 with ViT-G | FineAction detectors · FineAction | average mAP: 18.24 -> 17.20 | Hurt |
Surprising findings
- Global (temporal self-attention) propagation blocks benefit from fewer blocks, whereas local convolutional blocks benefit from more blocks.
- The 3D (spatiotemporal) clip-level propagation gave similar accuracy to the 1D temporal-only variant but used substantially more memory (27GB vs 9.3GB).
Limitations
- One-stage anchor-free heads suffer from background errors, requiring a better regression loss.
- Added propagation blocks increase trainable parameters and training memory, requiring checkpointing for larger configurations.
- Inference speed is slower than convolution-based TAD methods due to the ViT backbone.
Experimental setup
Datasets: THUMOS14 ActivityNet-1.3 FineAction Kinetics-400
Benchmarks: THUMOS14 ActivityNet-1.3 FineAction
Metrics: mAP average mAP over tIoU thresholds
Baselines: BasicTAD ActionFormer TriDet TALLFormer Re2TAL AFSD STPT VideoMAE V2 e2e-tadtr PBRNet
Builds on: VideoMAE ViTDet BasicTAD AFSD DETR
Closest papers by meaning
- Long-Short Temporal Contrastive Learning of Video Transformers (CVPR 2022 · similarity 0.89)
- Self-Supervised Video Transformer (CVPR 2022 · similarity 0.89)
- An Empirical Study of End-to-End Temporal Action Detection (CVPR 2022 · similarity 0.89)
- End-to-End Temporal Action Detection with 1B Parameters Across 1000 Frames (CVPR 2024 · similarity 0.87)
- TE-TAD: Towards Full End-to-End Temporal Action Detection via Time-Aligned Coordinate Expression (CVPR 2024 · similarity 0.87)
- Dual DETRs for Multi-Label Temporal Action Detection (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.