Research map / Video / Action recognition
Action Detail Matters: Refining Video Recognition with Local Action Queries
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FocusVideo, which adds learnable action queries and a shared-parameter local query streaming branch on top of a frozen CLIP video backbone with ST-Adapters, using a parameter-free cross-attention interaction to focus on action-relevant regions for video action recognition.
Problem
Video transformers capture global spatiotemporal context but attend to background noise and lack focused representations of action-relevant local details, without access to region-level labels.
Approach
Learnable action queries are propagated through an autoregressive-like shared query propagation layer and interact with video tokens via a parameter-free reuse of the video transformer's self-attention (treated as cross-attention for queries), with the local queries supervised by a reconstruction contrastive loss against offline-trained video features.
Key results
- FocusVideo-B/16 with 8 frames reaches 84.1% top-1 on Kinetics-400, a 2.1-point gain over the ST-Adapter global baseline (82.0%).
- FocusVideo-L/14 with 32 frames reaches 88.0% top-1 on Kinetics-400 and 72.9% on Something-Something-V2.
- Ablation shows the local branch adds about 1.7 points on K400 (82.4 to 84.1) with only about 8M extra learnable parameters on top of ST-Adapter.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add the QP Layer to the global video branch (ablation) | FocusVideo-B/16 · Kinetics-400 | Top-1 accuracy: 82.4 -> 83.5 | Helped |
| Add Seamless Feature Interaction (ablation) | FocusVideo-B/16 · Kinetics-400 | Top-1 accuracy: 83.5 -> 83.9 | Helped |
| Add Local Spatiotemporal Modeling (ablation) | FocusVideo-B/16 · Kinetics-400 | Top-1 accuracy: 83.9 -> 84.1 | Helped |
| Vary the number of action queries from 2 to 4, 8, or 16 (ablation) | FocusVideo-B/16 · Kinetics-400 | Top-1 accuracy: 2: 83.8; 4: 83.8; 8: 84.1; 16: 84.0 | Mixed |
| Use shared multi-scale query propagation across all layers rather than only top/half layers or non-shared parameters (ablation) | FocusVideo-B/16 · Kinetics-400 | Top-1 accuracy: Only top layer: 83.7; shared half layers: 83.9; shared all layers: 84.1; Non-shared all layers: 83.4 | Mixed |
| Use attention-only feature interaction instead of none or attention with T-AD/FFN (ablation) | FocusVideo-B/16 · Kinetics-400 | Top-1 accuracy: None: 83.8; Attention: 84.1; Attention + T-AD: 82.7; Attention + FFN: 83.2 | Mixed |
| Fuse global and local features by direct addition rather than concatenation or linear projection plus addition (ablation) | FocusVideo-B/16 · Kinetics-400 | Top-1 accuracy: Concatenation + Linear: 83.6; Add: 84.1; Linear + Add: 83.9 | Mixed |
| Use offline reconstruction supervision rather than no additional supervision, classification cross-entropy, or online reconstruction (ablation) | FocusVideo-B/16 · Kinetics-400 | Top-1 accuracy: None: 82.7; Cls Cross entropy: 83.5; Reconstruction (online): 83.4; Reconstruction (offline): 84.1 | Mixed |
| Add the QP Layer, Seamless Feature Interaction, and Local Spatiotemporal Modeling to the global branch (ablation) | FocusVideo-B/16 · Something-Something-V2 | Top-1 accuracy: 66.8 -> 67.7 -> 68.0 -> 68.4 | Helped |
| Vary the number of action queries from 2 to 4, 8, or 16 (ablation) | FocusVideo-B/16 · Something-Something-V2 | Top-1 accuracy: 2: 67.9; 4: 68.1; 8: 68.4; 16: 68.2 | Mixed |
| Use shared multi-scale query propagation across all layers rather than only top/half layers or non-shared parameters (ablation) | FocusVideo-B/16 · Something-Something-V2 | Top-1 accuracy: Only top layer: 67.7; shared half layers: 68.0; shared all layers: 68.4; Non-shared all layers: 66.5 | Mixed |
| Use attention-only feature interaction instead of none or attention with T-AD/FFN (ablation) | FocusVideo-B/16 · Something-Something-V2 | Top-1 accuracy: None: 68.0; Attention: 68.4; Attention + T-AD: 67.2; Attention + FFN: 67.7 | Mixed |
Surprising findings
- Non-shared query propagation parameters across layers hurt performance (83.4 vs 84.1 on K400), attributed to overfitting.
- Adding the temporal adapter or FFN to the feature interaction module reduces accuracy relative to attention-only reuse.
Limitations
- Relies on frozen CLIP backbones with only adapters and local branch learnable.
- Performance declines with too many action queries, attributed to introducing unnecessary regions.
- Best query supervision requires offline-trained video features as targets.
Experimental setup
Datasets: Kinetics-400 Something-Something-V2
Benchmarks: Kinetics-400 Something-Something-V2
Metrics: Top-1 accuracy Top-5 accuracy GFLOPs
Baselines: ST-Adapter EVL AIM ActionCLIP X-CLIP Vita-CLIP STAN M2-CLIP MoTE OST ILA DUALPATH
Builds on: ST-Adapter CLIP Video Transformer EVL
Closest papers by meaning
- Learning To Refactor Action and Co-Occurrence Features for Temporal Action Localization (CVPR 2022 · similarity 0.89)
- Large-Scale Weakly-Supervised Pre-Training for Video Action Recognition (CVPR 2019 · similarity 0.88)
- Pulling Actions out of Context: Explicit Separation for Effective Combination (CVPR 2018 · similarity 0.88)
- ActionAtlas: A VideoQA Benchmark for Domain-specialized Action Recognition (NeurIPS 2024 · similarity 0.87)
- FACT: Frame-Action Cross-Attention Temporal Modeling for Efficient Action Segmentation (CVPR 2024 · similarity 0.87)
- Unsupervised Action Segmentation via Fast Learning of Semantically Consistent Actoms (AAAI 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.