Research map / Video / Action recognition
Align Before Adapt: Leveraging Entity-to-Region Alignments for Generalizable Video Action Recognition
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes "Align before Adapt" (ALT), which aligns region-aware CLIP frame embeddings to an offline entity text corpus and uses the aligned entity embeddings as queries in a transformer video adapter for action recognition.
Problem
Existing CLIP-based video action recognition methods follow an "adapt then align" paradigm that aligns pooled video features only with action label text, overlooking entity-level grounding needed for complex activities and generalization to unseen classes.
Approach
Before adapting to video, match token-merged region embeddings of each frame to an LLM/WordNet-built entity text corpus via Gumbel-softmax with a one-hot straight-through trick, then use the aligned entity text embeddings as queries in a transformer video adapter, supervised by action-label text embeddings.
Key results
- ALT-L/14 reaches 88.1% top-1 on Kinetics-400 at 4947 GFLOPs (32 frames), 0.4% above AIM-L/14 with 55% fewer GFLOPs.
- In 2-shot HMDB-51, ALT-B/16 reaches 64.3% top-1, 7.1% above prior image-language SOTA.
- Zero-shot on HMDB-51, ALT-B/16 reaches 52.9% vs ViFi-CLIP 51.3%; on UCF-101 ALT-B/16 reaches 79.4% vs 76.8%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add the entity text corpus instead of using action-name text embeddings (ablation) | ALT video adapter · HMDB-51 2-shot | top-1 accuracy: 55.2 -> 58.1 | Helped |
| Add entity-to-region alignment using region-aware visual embeddings rather than global visual embeddings (ablation) | ALT · HMDB-51 2-shot | top-1 accuracy: 58.1 -> 64.3 | Helped |
| Add the video adapter that uses text embeddings to guide adaptation (ablation) | ALT · Kinetics-400, 8 frames | top-1 accuracy: 81.7 -> 82.0 | Helped |
| Replace four self-attention layers with the full video adapter, including cross-attention (ablation) | ALT · Kinetics-400, 8 frames | top-1 accuracy: 81.1 -> 82.8 | Helped |
| Add cross-attention to the video adapter (ablation) | ALT · HMDB-51 2-shot | top-1 accuracy: 55.0 -> 61.3 | Helped |
| Add the 1D temporal convolution module to the video adapter (ablation) | ALT · Kinetics-400, 8 frames | top-1 accuracy: 81.8 -> 82.5 | Helped |
| Add self-attention to the cross-attention and 1D-convolution video adapter (ablation) | ALT · Kinetics-400, 8 frames | top-1 accuracy: 82.3 -> 82.8 | Helped |
| Freeze the image encoder rather than fine-tune it (ablation) | ALT · Kinetics-400 | top-1 accuracy: 82.8 -> 81.4 | Hurt |
| Add spatial, MLP, and temporal adapters to the frozen image encoder (ablation) | ALT · Kinetics-400 | top-1 accuracy: 81.4 -> 82.5 | Helped |
| Use STM adapters rather than full image-encoder fine-tuning (ablation) | ALT · Kinetics-400 | GFLOPs: 163 vs. 110 | Hurt |
| Add ten learnable tokens to each image and text encoder layer while freezing the backbone (ablation) | ALT few-shot model · HMDB-51 2-shot | top-1 accuracy: 64.3 -> 65.3 | Helped |
| Fine-tune the full visual branch rather than using STM adapters (ablation) | ALT few-shot model · HMDB-51 2-shot | top-1 accuracy: 58.1 -> 64.3 | Helped |
Surprising findings
- Linear-probed ALT-B/16 on UCF-101 and HMDB-51 is comparable to fully trained VideoMAE.
Limitations
- Few-shot and zero-shot gains depend on an entity corpus generated via LLM and WordNet pipelines, whose quality is not systematically evaluated.
- Relies on large-scale CLIP-400M pretraining; the entity-region alignment depends on CLIP's region-level embedding quality.
Experimental setup
Datasets: Kinetics-400 Kinetics-600 UCF-101 HMDB-51 action-label-derived text corpus (LLM and WordNet generated)
Benchmarks: Kinetics-400 Kinetics-600 HMDB-51 UCF-101
Metrics: top-1 accuracy top-5 accuracy GFLOPs few-shot top-1 accuracy
Baselines: X-CLIP-B/16 ViFi-CLIP ActionCLIP AIM EVL ST-Adapter MTV-H VideoSwin TimeSformer ViViT-H/16
Builds on: CLIP ToMe X-CLIP AIM ViFi-CLIP
Closest papers by meaning
- Temporal Alignment-Free Video Matching for Few-shot Action Recognition (CVPR 2025 · similarity 0.89)
- Efficient Transfer Learning for Video-language Foundation Models (CVPR 2025 · similarity 0.89)
- TOPA: Extending Large Language Models for Video Understanding via Text-Only Pre-Alignment (NeurIPS 2024 · similarity 0.88)
- VidLA: Video-Language Alignment at Scale (CVPR 2024 · similarity 0.88)
- Storyboard-guided Alignment for Fine-grained Video Action Recognition (NeurIPS 2025 · similarity 0.88)
- OST: Refining Text Knowledge with Optimal Spatio-Temporal Descriptor for General Video Recognition (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.