Research map / Video / Action recognition
Temporal Alignment-Free Video Matching for Few-shot Action Recognition
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TEAM, a temporal alignment-free few-shot action recognition method that represents videos with learnable pattern tokens compared token-wise, plus a support-set adaptation that removes information shared across classes.
Problem
Few-shot action recognition needs robust video matching, but frame- and tuple-level alignment methods rely on pre-defined, length-dependent units, limiting flexibility for varying action lengths and speeds, and incur quadratic matching cost.
Approach
Represent each video by a fixed set of learnable pattern tokens (instance and exclusive, via cross-attention add/subtract) and compare videos token-wise without any temporal alignment; adapt support tokens per episode by removing information shared with other classes.
Key results
- TEAM with ResNet reaches 62.8 (HMDB51), 78.4 (Kinetics), 75.1 (UCF101) 1-shot accuracy, state of the art on most settings.
- With ViT backbone, TEAM reaches 70.9 (HMDB51 1-shot), 85.5 (Kinetics 1-shot), 94.5 (UCF101 1-shot) and surpasses all compared methods in every case.
- Matching time grows linearly with frame count versus quadratic for alignment methods, and TEAM has the fewest parameters except OTAM.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use TEAM instead of the strongest listed baseline | TEAM with ResNet · HMDB51 5-way 1-shot | accuracy: 60.8 -> 62.8 | Helped |
| Use TEAM instead of the strongest listed baseline | TEAM with ResNet · Kinetics 5-way 1-shot | accuracy: 74.9 -> 75.1 | Helped |
| Use TEAM instead of the strongest listed baseline | TEAM with ResNet · UCF101 5-way 5-shot | accuracy: 97.1 -> 96.2 | Hurt |
| Use TEAM instead of the strongest listed baseline | TEAM with ViT · HMDB51 5-way 5-shot | accuracy: 83.2 -> 85.5 | Helped |
| Use TEAM instead of the strongest listed baseline | TEAM with ViT · UCF101 5-way 1-shot | accuracy: 93.6 -> 94.5 | Helped |
| Use TEAM instead of the strongest listed baseline | TEAM with ViT · SSv2-Small 5-way 1-shot | accuracy: 47.9 -> 47.2 | Hurt |
| Use TEAM instead of the strongest listed baseline | TEAM with ViT · HMDB51 5-way 5-shot cross-domain | accuracy: 78.5 -> 81.1 | Helped |
| Add exclusive pattern tokens (ablation) | TEAM instance-token model · HMDB51, Kinetics, and UCF101 5-way 1-shot | accuracy: 61.8 -> 62.5; 74.6 -> 75.0; 86.7 -> 86.8 | Helped |
| Add support-token adaptation without entanglement management (ablation) | TEAM with instance and exclusive tokens · HMDB51, Kinetics, and UCF101 5-way 1-shot | accuracy: 62.5 -> 62.2; 75.0 -> 74.8; 86.8 -> 87.1 | Mixed |
| Add entanglement management to support-token adaptation (ablation) | TEAM with instance, exclusive, and adaptive tokens · HMDB51, Kinetics, and UCF101 5-way 1-shot | accuracy: 62.2 -> 62.8; 74.8 -> 75.1; 87.1 -> 87.2 | Helped |
| Vary the number of pattern tokens (ablation) | TEAM · HMDB51, Kinetics, and UCF101 | accuracy | No clear effect |
Surprising findings
- Removing the adaptation's magnitude control (variant c) slightly lowers accuracy versus no entanglement term, indicating adaptation strength must be managed.
Limitations
- Evaluated mainly on trimmed, short-term video benchmarks, where alignment cost is less critical.
- Pattern token count is a hyperparameter; optimal value (60) appears dataset-agnostic only empirically.
Experimental setup
Datasets: HMDB51 Kinetics UCF101 SSv2-Small
Benchmarks: HMDB51 Kinetics UCF101 SSv2-Small
Metrics: accuracy (1-shot, 5-shot) computational time (ms) FLOPs parameter count
Baselines: OTAM TRX STRM HyRSM MoLo TATs ProtoNet SloshNet TA2N MTFAN
Builds on: OTAM TRX STRM HyRSM MoLo
Closest papers by meaning
- Searching for Better Spatio-temporal Alignment in Few-Shot Action Recognition (NeurIPS 2022 · similarity 0.91)
- TA2N: Two-Stage Action Alignment Network for Few-Shot Action Recognition (AAAI 2022 · similarity 0.91)
- TF-CADE: Foreground-Concentrated Text-Video Alignment for Zero-Shot Temporal Action Detection (CVPR 2026 · similarity 0.90)
- Align Before Adapt: Leveraging Entity-to-Region Alignments for Generalizable Video Action Recognition (CVPR 2024 · similarity 0.89)
- Storyboard-guided Alignment for Fine-grained Video Action Recognition (NeurIPS 2025 · similarity 0.89)
- Frame Order Matters: A Temporal Sequence-Aware Model for Few-Shot Action Recognition (AAAI 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.