AI Paper Atlas

Research map / Video / Action recognition

Optimizing Factorized Encoder Models: Time and Memory Reduction for Scalable and Efficient Action Recognition

Shreyank N Gowda, Anurag Arnab, Jonathan Huang

ECCV 2024 · Video › Action recognition · approach: Spatiotemporal CNNs for action recognition · 2 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
freeze the spatial transformer naively (no adapter, temporal from scratch) (ablation)ViViT-B · Something-something v2top-1 accuracy: 64.45 -> 27.75Hurt
add a lightweight adapter after the frozen spatial transformer (ablation)ViViT-B with frozen spatial · Something-something v2top-1 accuracy: 27.75 -> 38.77Improved
initialize temporal transformer from an 8-frame ViViT checkpoint (ablation)ViViT-B with frozen spatial and adapter · Something-something v2top-1 accuracy: 38.77 -> 63.85Improved
initialize temporal transformer from VideoMAE instead of ViViT-8f (ablation)ViViT-B with frozen spatial and adapter · Something-something v2top-1 accuracy: 63.85 -> 58.54; train time 0.62x -> 2.51xHurt
keep the spatial transformer unfrozen in stage 2 (ablation)SFA ViViT-B · Something-something v2top-1 accuracy: 63.85 -> 64.13; train time 0.62x -> 1.37xNo clear effect
use 8 or more frames for stage 1 pretraining rather than fewer or image-only init (ablation)SFA ViViT · Kinetics-400top-1 accuracyImproved
initialize 8-frame model on a different dataset than the fine-tuning dataset (ablation)SFA ViViT · Kinetics-400top-1/top-5 accuracy: 82.81/93.98 -> 76.79/92.35Hurt
apply SFA training to ViViT-L/HViViT · Kinetics-400top-1 accuracy and train time: 82.59 -> 82.78 (L), 84.21 -> 84.42 (H); 0.56x/0.57x timeImproved
apply SFA training to ViViT-HViViT-H 96f · Moments in Timetop-1 accuracy: 38.17 -> 39.96Improved
apply SFA training to ViViT-LViViT-L 128f · Something-something v2top-1 accuracy: 64.45 -> 63.85Hurt
apply SFA staged training to Dualformer, VideoSwin, Video-FocalNetshierarchical video models · Kinetics-400top-1 accuracy and train time: about 0.51x-0.59x training time; Video-FocalNet-B 83.55 -> 84.25Improved
freeze the spatial transformer to enable a larger ViT-g backbone and more framesSFA ViViT-g · Kinetics-400max frames before OOM: 8 -> 48 framesImproved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy