Research map / Video / Action recognition
End-to-End Spatio-Temporal Action Localisation with Video Transformers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes STAR, a fully end-to-end DETR-style video transformer that outputs spatio-temporal tubelets from a clip with factorised queries and factorised decoder attention, trainable with sparse keyframe or full tube supervision.
Problem
Spatio-temporal action localisation typically relies on external person proposals, memory banks, and NMS; proposal-free tubelet models have lagged behind. The paper asks whether an end-to-end transformer can outperform two-stage approaches and predict tubelets even with sparse keyframe supervision.
Approach
Adapts DETR to video with factorised learned queries (spatial plus temporal embeddings), a decoder that uses factorised spatial-then-temporal self-attention and per-frame cross-attention, and per-frame or tubelet-level Hungarian matching so that queries bind to persons across frames.
Key results
- STAR/L reaches 45.1 Frame AP on AVA-Kinetics (with AVA-K training), outperforming InternVideo and VideoMAE v2 without test-time augmentation.
- On UCF101-24, STAR/CSN-152 improves Video AP50 over TubeR by 7.0 points at the same backbone, and STAR/L surpasses the prior best Video AP50 by 13.2 points.
- With only one labelled frame per UCF101-24 video, the model still reaches vAP20 of 77.1, and one frame per 24 labelled frames gives within 0.7 points of full supervision.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Bind each query to a person rather than to an action (ablation) | STAR · AVA | mAP: 23.6 -> 26.7; 25.7 -> 27.8 | Helped |
| Use factorised spatial and temporal queries (ablation) | STAR · AVA and UCF101-24 | fAP / vAP50: 25.2 -> 26.3; 59.5 -> 63.4 | Helped |
| Use tubelet matching instead of per-frame matching (ablation) | STAR · UCF101-24 | vAP50: 63.5 -> 63.4 | No clear effect |
| Use tubelet matching instead of per-frame matching (ablation) | STAR · AVA | fAP: 26.7 -> 26.3 | Hurt |
| Subsample training labels to one frame per video (ablation) | STAR · UCF101-24 | vAP50: 63.4 -> 48.5 | Hurt |
| Increase the decoder depth from 0 to 6 layers (ablation) | STAR · AVA | mAP: 23.4 -> 26.5 | Helped |
| Use factorised attention instead of full attention (ablation) | STAR · AVA | mAP: 26.4 -> 26.7 | Helped |
| Increase input resolution (ablation) | STAR · AVA | mAP: 25.4 -> 30.0 | Helped |
| Use CLIP pretraining instead of ImageNet-21K pretraining (ablation) | STAR/B and STAR/L · AVA | mAP: 26.7 -> 30.3; 27.0 -> 36.2 | Helped |
| Use STAR/L instead of the strongest reported baseline | STAR/L · AVA-Kinetics | mAP: 43.9 -> 45.1 | Helped |
| Use STAR/L instead of the strongest reported baseline | STAR/L · MultiSports | fAP: 40.8 -> 59.3 | Helped |
Surprising findings
- Using only one labelled frame per video on UCF101-24 still yields meaningful tubelets (vAP20 77.1), suggesting redundancy and temporal inductive biases allow sparse supervision.
- The authors could not reproduce TubeR's reported results with the public code (20 mAP vs 31.1 reported at 256p).
Limitations
- Relies on linking tubelets with a causal algorithm for long videos such as UCF101-24, which are longer than the 32-frame input window.
- Best AVA results require training on AVA-Kinetics and evaluating on AVA, and large ViViT backbones pretrained on CLIP and K700 are needed for top numbers.
- Tubelet matching trades off per-frame Frame AP against Video AP.
Experimental setup
Datasets: AVA v2.2 AVA-Kinetics UCF101-24 JHMDB51-21 MultiSports Kinetics 400 Kinetics 700 ImageNet-21K CLIP pretraining
Benchmarks: AVA v2.2 AVA-Kinetics UCF101-24 JHMDB51-21 MultiSports
Metrics: Frame AP (fAP) mAP Video AP (vAP20) Video AP (vAP50) Video AP (vAP50:95)
Baselines: TubeR ACAR MeMViT VideoMAE v2 InternVideo WOO Unified HIT FBIL MOC ACT
Builds on: DETR ViViT TubeR
Closest papers by meaning
- TubeR: Tubelet Transformer for Video Action Detection (CVPR 2022 · similarity 0.90)
- TubeDETR: Spatio-Temporal Video Grounding With Transformers (CVPR 2022 · similarity 0.88)
- Few-Shot Transformation of Common Actions Into Time and Space (CVPR 2021 · similarity 0.87)
- Temporal Action Localization by Structured Maximal Sums (CVPR 2017 · similarity 0.87)
- Improving Action Localization by Progressive Cross-Stream Cooperation (CVPR 2019 · similarity 0.87)
- Bridge the Gap: From Weak to Full Supervision for Temporal Action Localization with PseudoFormer (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.