Research map / Architectures and efficiency / Attention and transformers
Multiscale Vision Transformers Meet Bipartite Matching for Efficient Single-stage Action Localization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Trains a plain vision transformer (MViTv2-S, ViT-B) end-to-end for action localization by applying a DETR-style bipartite matching loss directly to output tokens, removing the external person detector, learnable queries and decoder.
Problem
Spatio-temporal action localization (joint person detection and action classification) is typically two-stage with an off-the-shelf detector, or single-stage with heavy DETR encoder-decoder heads; both are costly or hard to train.
Approach
Apply the OWL-ViT-style bipartite (Hungarian) matching loss directly to the spatio-temporal output tokens of a video transformer, with simple MLP heads for box, actor likelihood and action classes, so no learnable queries or decoder are needed; token selection (central frame for detection, temporal pooling for classification) pairs tokens across heads.
Key results
- BMViT with MViTv2-S (16x4, 256px, K400) reaches 30.0 mAP on AVA 2.2 at 121 GFLOPs, versus 26.8 mAP for two-stage MViTv2-S with ROI align at 65+246 GFLOPs.
- With ViT-B (K400+MAE) at 288px, BMViT achieves 31.4 mAP on AVA 2.2 at 350 GFLOPs, compared with EVAD's 32.1 at 425 GFLOPs.
- On UCF101-24, BMViT with ViT-B pre-trained on K710 reaches 90.7 mAP, and on JHMDB51-21 it reaches 88.4 mAP.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train MViTv2-S with bipartite matching instead of RoI align on precomputed boxes (ablation) | MViTv2-S on AVA 2.2 · AVA 2.2 | mAP: 26.8 -> 30.0 | Helped |
| Remove the last pooling layer (ablation) | BMViT with MViTv2-S on AVA 2.2 · AVA 2.2 | mAP: 27.5 -> 30.0 | Helped |
| Use both central-frame tokens for detection and independently pooled past/future tokens for classification (2(C+T)) (ablation) | BMViT on AVA 2.2 · AVA 2.2 | mAP: 29.1 -> 30.0 | Helped |
| Use all tokens without reduction (Singleton) rather than 2(C+T) token selection (ablation) | BMViT on AVA 2.2 · AVA 2.2 | mAP: 30.0 -> 28.5 | Hurt |
| Use tubelet temporal pooling rather than 2(C+T) token selection (ablation) | BMViT on AVA 2.2 · AVA 2.2 | mAP: 30.0 -> 28.7 | Hurt |
| Use max pooling over time for action tokens rather than 2(C+T) token selection (ablation) | BMViT on AVA 2.2 · AVA 2.2 | mAP: 30.0 -> 28.3 | Hurt |
| Keep a variable input aspect ratio rather than resizing to a fixed aspect ratio (ablation) | BMViT on AVA 2.2 · AVA 2.2 | mAP: 30.0 -> 28.5 | Hurt |
Surprising findings
- Fixed aspect-ratio resizing matters: a variable number of tokens during training drops mAP from 30.0 to 28.5 and hampers convergence.
- Scaling to Hiera-L in end-to-end single-stage training gives 38.5 mAP, below the two-stage Hiera-L at 39.8 mAP.
Limitations
- Performance drops with variable-aspect-ratio training, so fixed-size inputs are required.
- Sub-par results when scaling to large backbones such as Hiera-L in end-to-end training.
- Accuracy on JHMDB51-21 lags behind EVAD despite high precision and recall.
Experimental setup
Datasets: AVA 2.2 UCF101-24 JHMDB51-21 Kinetics-400
Benchmarks: AVA 2.2 UCF101-24 JHMDB51-21
Metrics: mAP
Baselines: MViTv2-S with ROI align TubeR STMixer EVAD MeMViT WOO ACAR-Net VideoMAE Hiera-L two-stage
Builds on: DETR OWL-ViT MViTv2 VideoMAE
Closest papers by meaning
- MViTv2: Improved Multiscale Vision Transformers for Classification and Detection (CVPR 2022 · similarity 0.86)
- Video Action Transformer Network (CVPR 2019 · similarity 0.85)
- Adapting Short-Term Transformers for Action Detection in Untrimmed Videos (CVPR 2024 · similarity 0.85)
- End-to-End Spatio-Temporal Action Localisation with Video Transformers (CVPR 2024 · similarity 0.85)
- Mr. DETR: Instructive Multi-Route Training for Detection Transformers (CVPR 2025 · similarity 0.84)
- DVANet: Disentangling View and Action Features for Multi-View Action Recognition (AAAI 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.