AI Paper Atlas

Research map / Video / Action recognition

Eventful Transformers: Leveraging Temporal Redundancy in Vision Transformers

Matthew Dutson, Yin Li, Mohit Gupta

ICCV 2023 · Video › Action recognition · approach: Contrastive video representation learning · 24 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
add temporal redundancy awareness (token gating, r=768) to ViTDetViTDet on VID, 1024 · ImageNet VID valmAP50 / GFlops: 82.93 -> 81.25 mAP50; 467.4 -> 122.3 GFlopsImproved
use reference-based gating instead of STGT lossy gating (ablation)Eventful ViTDet · ImageNet VID val (1024, r=256)mAP50: 68.73 -> 73.29Improved
extend redundancy awareness to query-key and attention-value products (ablation)Eventful ViTDet · ImageNet VID val (1024, r=256)GFlops: 142.3 -> 53.3Improved
extend redundancy awareness to attention products at low r (accuracy side) (ablation)Eventful ViTDet · ImageNet VID val (1024, r=256)mAP50: 76.60 -> 73.29Hurt
use threshold policy instead of top-r policy (ablation)Eventful ViTDet · ImageNet VID val (1024)mAP50 vs GFlops: h=5.0: 78.11 at 133.5 GFlops vs r=512: 79.38 at 87.8 GFlopsHurt
add spatial pooling of k and v to the temporal model (ablation)Eventful ViTDet · ImageNet VID val (1024)GFlops: 388.1 -> 217.0 GFlops at r=2048 (80.15 -> 80.14 mAP50)Improved
naively replace ViViT spatial model with Eventful version without fine-tuning temporal model (ablation)ViViT on EPIC-Kitchens · EPIC-Kitchens 100accuracy: about -10% at r=100Hurt
fine-tune non-Eventful temporal model on outputs of frozen Eventful spatial model (ablation)Eventful ViViT · EPIC-Kitchens 100accuracy: reduction of 1.62% at r=140 with 2.4x lower costImproved
apply Eventful conversion to ViViTViViT on Kinetics-400 · Kinetics-400top-1 accuracy: 79.06 -> 77.62; 3.360 -> 1.814 TFlopsMixed
lower the policy budget r in Eventful ViTDet (1024) (ablation)Eventful ViTDet · ImageNet VID valmAP50: 82.94 (r=2048) -> 73.29 (r=256)Hurt
add temporal redundancy awareness to ViTDet (GPU runtime)ViTDet 1024 · GPU runtimems: 86.6 -> 69.9 (temporal); 86.6 -> 38.1 (spatiotemporal)Improved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy