Research map / Video / Action recognition
Optimizing Factorized Encoder Models: Time and Memory Reduction for Scalable and Efficient Action Recognition
Shreyank N Gowda, Anurag Arnab, Jonathan Huang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| freeze the spatial transformer naively (no adapter, temporal from scratch) (ablation) | ViViT-B · Something-something v2 | top-1 accuracy: 64.45 -> 27.75 | Hurt |
| add a lightweight adapter after the frozen spatial transformer (ablation) | ViViT-B with frozen spatial · Something-something v2 | top-1 accuracy: 27.75 -> 38.77 | Improved |
| initialize temporal transformer from an 8-frame ViViT checkpoint (ablation) | ViViT-B with frozen spatial and adapter · Something-something v2 | top-1 accuracy: 38.77 -> 63.85 | Improved |
| initialize temporal transformer from VideoMAE instead of ViViT-8f (ablation) | ViViT-B with frozen spatial and adapter · Something-something v2 | top-1 accuracy: 63.85 -> 58.54; train time 0.62x -> 2.51x | Hurt |
| keep the spatial transformer unfrozen in stage 2 (ablation) | SFA ViViT-B · Something-something v2 | top-1 accuracy: 63.85 -> 64.13; train time 0.62x -> 1.37x | No clear effect |
| use 8 or more frames for stage 1 pretraining rather than fewer or image-only init (ablation) | SFA ViViT · Kinetics-400 | top-1 accuracy | Improved |
| initialize 8-frame model on a different dataset than the fine-tuning dataset (ablation) | SFA ViViT · Kinetics-400 | top-1/top-5 accuracy: 82.81/93.98 -> 76.79/92.35 | Hurt |
| apply SFA training to ViViT-L/H | ViViT · Kinetics-400 | top-1 accuracy and train time: 82.59 -> 82.78 (L), 84.21 -> 84.42 (H); 0.56x/0.57x time | Improved |
| apply SFA training to ViViT-H | ViViT-H 96f · Moments in Time | top-1 accuracy: 38.17 -> 39.96 | Improved |
| apply SFA training to ViViT-L | ViViT-L 128f · Something-something v2 | top-1 accuracy: 64.45 -> 63.85 | Hurt |
| apply SFA staged training to Dualformer, VideoSwin, Video-FocalNets | hierarchical video models · Kinetics-400 | top-1 accuracy and train time: about 0.51x-0.59x training time; Video-FocalNet-B 83.55 -> 84.25 | Improved |
| freeze the spatial transformer to enable a larger ViT-g backbone and more frames | SFA ViViT-g · Kinetics-400 | max frames before OOM: 8 -> 48 frames | Improved |
Closest papers by meaning
- ViViT: A Video Vision Transformer (ICCV 2021 · similarity 0.91)
- Video Transformers: A Survey (TPAMI 2023 · similarity 0.90)
- Adapting Short-Term Transformers for Action Detection in Untrimmed Videos (CVPR 2024 · similarity 0.89)
- Perception Encoder: The best visual embeddings are not at the output of the network (NeurIPS 2025 · similarity 0.87)
- DVANet: Disentangling View and Action Features for Multi-View Action Recognition (AAAI 2024 · similarity 0.87)
- TrAction: Action Recognition with Sparse Trajectories (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
