Research map / Video / Video segmentation and tracking
Implicit Temporal Modeling with Learnable Alignment for Video Recognition
Shuyuan Tu, Qi Dai, Zuxuan Wu, Zhi-Qi Cheng, Han Hu, Yu-Gang Jiang
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add implicit learnable alignment to CLIP without temporal modeling (ablation) | CLIP ViT-B/32 · Kinetics-400 | top-1 accuracy: 79.8 -> 81.3 | Improved |
| use learned alignment mask before pooling the MI token instead of plain average pooling (ablation) | ILA · Kinetics-400 | top-1 accuracy: 80.4 -> 81.3 | Improved |
| align adjacent frames instead of anchoring to first or middle frame (ablation) | ILA · Kinetics-400 | top-1 accuracy: 80.7 / 80.8 -> 81.3 | Improved |
| use 2D convolution instead of window attention to predict interactive points (ablation) | ILA · Kinetics-400 | top-1 accuracy: 80.8 -> 81.3; FLOPs 114G -> 40G | Improved |
| pool aligned features into a single MI token instead of element-wise addition or direct concatenation (ablation) | ILA · Kinetics-400 | top-1 accuracy: 80.2 / 80.6 -> 81.3 | Improved |
| insert ILA into every block instead of one group of blocks (ablation) | ILA · Kinetics-400 | top-1 accuracy: 80.3-80.5 -> 81.3 | Improved |
| insert ILA into shallow blocks rather than deep blocks (ablation) | ILA · Kinetics-400 | top-1 accuracy: 80.4 (blocks 1-3) vs 80.3 (blocks 10-12) | No clear effect |
| plug ILA into EVL (CLIP-based) | EVL-ViT-B/16 · Kinetics-400 | top-1 accuracy: 82.9 -> 83.5 | Improved |
| replace temporal attention in TimeSformer with ILA | TimeSformer-ViT-B/16 · Kinetics-400 | top-1 accuracy: 78.0 -> 79.8; FLOPs 196G -> 164G | Improved |
| use ILA instead of Swin under the same IN-21K init without CLIP text | ViT-B/16 IN-21K · Kinetics-400 | top-1 accuracy: 82.7 -> 83.3 | Improved |
| add video-specific text representation tuning (ablation) | ILA · Kinetics-400 (ViT-B/32, 8f) | top-1 accuracy: 80.8 -> 81.3 | Improved |
| add two extra 3x3 convolution layers in alignment module for SSv2 (ablation) | ILA · Something-Something-V2 | top-1 accuracy: +0.6% top-1 | Improved |
Closest papers by meaning
- SmartCLIP: Modular Vision-language Alignment with Identification Guarantees (CVPR 2025 · similarity 0.86)
- TOPA: Extending Large Language Models for Video Understanding via Text-Only Pre-Alignment (NeurIPS 2024 · similarity 0.86)
- Structured Video-Language Modeling with Temporal Grouping and Spatial Grounding (ICLR 2024 · similarity 0.86)
- SVLTA: Benchmarking Vision-Language Temporal Alignment via Synthetic Video Situation (CVPR 2025 · similarity 0.86)
- STOP: Integrated Spatial-Temporal Dynamic Prompting for Video Understanding (CVPR 2025 · similarity 0.86)
- Kronecker Mask and Interpretive Prompts are Language-Action Video Learners (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
