Research map / Video / Video-language understanding
Enhanced Motion-Text Alignment for Image-to-Video Transfer Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MoTED, which uses GPT-4-generated conceptual and discriminative motion descriptions plus a temporal motion encoder and text motion adapter to symmetrically align motion features with CLIP for video recognition.
Problem
CLIP-based video recognition methods add temporal modules only to the visual encoder, while text inputs remain category names with no motion information, leaving the text side weak for motion understanding and creating a visual-textual asymmetry.
Approach
Generate motion-enhanced text descriptions with GPT-4 (concept descriptions and top-k discriminative descriptions contrasting similar classes), then build a parallel temporal motion encoder on the visual side and a text motion adapter on the textual side, aligning both with contrastive losses.
Key results
- Fully supervised: MoTED (ViT-L/14, 32 frames) reaches 73.8% top-1 on SSv2 and 88.8% on Kinetics-400, versus DiST's 73.1% and 88.0%.
- Zero-shot (trained on K400): MoTED (ViT-B/16) achieves 58.2% on HMDB51, 78.3% on UCF101 and 69.9% on K600, improving over the prior best by +3.8, +6.0 and +2.5 points.
- Ablation on SSv2: adding both concept and discriminative descriptions raises accuracy from 65.7% (class names) to 70.1%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add conceptual descriptions to category names (ablation) | MoTED · Something-Something-V2 and Kinetics-400 validation | Top-1 accuracy: 65.7 -> 66.0; 82.9 -> 84.5 | Mixed |
| Add discriminative descriptions to category names (ablation) | MoTED · Something-Something-V2 and Kinetics-400 validation | Top-1 accuracy: 65.7 -> 69.9; 82.9 -> 84.0 | Helped |
| Use both conceptual and discriminative descriptions (ablation) | MoTED · Something-Something-V2 and Kinetics-400 validation | Top-1 accuracy: 65.7 -> 70.1; 82.9 -> 85.1 | Helped |
| Replace serial motion modules with a parallel motion encoder (ablation) | MoTED · Something-Something-V2 and Kinetics-400 validation | Top-1 accuracy: 64.3 -> 68.6; 81.5 -> 83.4 | Helped |
| Add a text motion adapter (ablation) | MoTED with parallel motion encoder · Something-Something-V2 and Kinetics-400 validation | Top-1 accuracy: 68.6 -> 70.1; 83.4 -> 85.1 | Helped |
| Add textual feature fusion (ablation) | MoTED · Something-Something-V2 and Kinetics-400 validation | Top-1 accuracy: 68.3 -> 68.9; 83.6 -> 84.2 | Helped |
| Add visual feature fusion (ablation) | MoTED · Something-Something-V2 and Kinetics-400 validation | Top-1 accuracy: 68.3 -> 69.5; 83.6 -> 84.5 | Helped |
| Use both textual and visual feature fusion (ablation) | MoTED · Something-Something-V2 and Kinetics-400 validation | Top-1 accuracy: +1.8%/+1.5% on SSv2/K400 | Helped |
| Increase input frames from 8 to 16 (ablation) | MoTED, ViT-B/16 · Something-Something-V2 validation | Top-1 accuracy: 70.1 -> 71.2 | Helped |
| Increase input frames from 16 to 32 (ablation) | MoTED, ViT-B/16 · Something-Something-V2 validation | Top-1 accuracy: 71.2 -> 71.9 | Helped |
| Increase input frames from 8 to 32 (ablation) | MoTED, ViT-B/16 · Kinetics-400 validation | Top-1 accuracy: 85.1 -> 86.2 | Helped |
Surprising findings
- Concept-only descriptions give only marginal gains on SSv2 (+0.3%), while discriminative descriptions contrasting similar classes give +4.2%, indicating they matter more for fine-grained motion.
Limitations
- Ablations are run only on ViT-B/16 with 8 frames at low sampling rate.
- Depends on GPT-4 generated descriptions; the generation process is not evaluated for quality or accuracy.
Experimental setup
Datasets: Kinetics-400 Something-Something-V2 Kinetics-600 HMDB51 UCF101
Benchmarks: Kinetics-400 Something-Something-V2 Kinetics-600 HMDB51 UCF101
Metrics: top-1 accuracy top-5 accuracy
Baselines: EVL ST-Adapter DiST X-CLIP Vita-CLIP ActionCLIP UniFormerV2 TAdaFormer
Builds on: CLIP X-CLIP DiST EVL ST-Adapter
Closest papers by meaning
- Enhancing Motion in Text-to-Video Generation with Decomposed Encoding and Conditioning (NeurIPS 2024 · similarity 0.87)
- OST: Refining Text Knowledge with Optimal Spatio-Temporal Descriptor for General Video Recognition (CVPR 2024 · similarity 0.87)
- Kronecker Mask and Interpretive Prompts are Language-Action Video Learners (ICLR 2025 · similarity 0.87)
- Tri-Modal Motion Retrieval by Learning a Joint Embedding Space (CVPR 2024 · similarity 0.86)
- TOPA: Extending Large Language Models for Video Understanding via Text-Only Pre-Alignment (NeurIPS 2024 · similarity 0.86)
- LaMP: Language-Motion Pretraining for Motion Generation, Retrieval, and Captioning (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.