AI Paper Atlas

Research map / Video / Video segmentation and tracking

Implicit Temporal Modeling with Learnable Alignment for Video Recognition

Shuyuan Tu, Qi Dai, Zuxuan Wu, Zhi-Qi Cheng, Han Hu, Yu-Gang Jiang

ICCV 2023 · Video › Video segmentation and tracking · approach: Self-supervised video representation learning · 53 citations (OpenAlex)

Read the paper

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
add implicit learnable alignment to CLIP without temporal modeling (ablation)CLIP ViT-B/32 · Kinetics-400top-1 accuracy: 79.8 -> 81.3Improved
use learned alignment mask before pooling the MI token instead of plain average pooling (ablation)ILA · Kinetics-400top-1 accuracy: 80.4 -> 81.3Improved
align adjacent frames instead of anchoring to first or middle frame (ablation)ILA · Kinetics-400top-1 accuracy: 80.7 / 80.8 -> 81.3Improved
use 2D convolution instead of window attention to predict interactive points (ablation)ILA · Kinetics-400top-1 accuracy: 80.8 -> 81.3; FLOPs 114G -> 40GImproved
pool aligned features into a single MI token instead of element-wise addition or direct concatenation (ablation)ILA · Kinetics-400top-1 accuracy: 80.2 / 80.6 -> 81.3Improved
insert ILA into every block instead of one group of blocks (ablation)ILA · Kinetics-400top-1 accuracy: 80.3-80.5 -> 81.3Improved
insert ILA into shallow blocks rather than deep blocks (ablation)ILA · Kinetics-400top-1 accuracy: 80.4 (blocks 1-3) vs 80.3 (blocks 10-12)No clear effect
plug ILA into EVL (CLIP-based)EVL-ViT-B/16 · Kinetics-400top-1 accuracy: 82.9 -> 83.5Improved
replace temporal attention in TimeSformer with ILATimeSformer-ViT-B/16 · Kinetics-400top-1 accuracy: 78.0 -> 79.8; FLOPs 196G -> 164GImproved
use ILA instead of Swin under the same IN-21K init without CLIP textViT-B/16 IN-21K · Kinetics-400top-1 accuracy: 82.7 -> 83.3Improved
add video-specific text representation tuning (ablation)ILA · Kinetics-400 (ViT-B/32, 8f)top-1 accuracy: 80.8 -> 81.3Improved
add two extra 3x3 convolution layers in alignment module for SSv2 (ablation)ILA · Something-Something-V2top-1 accuracy: +0.6% top-1Improved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy