Research map / Video / Action recognition
Frame Order Matters: A Temporal Sequence-Aware Model for Few-Shot Action Recognition
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TSAM, a few-shot action recognition model that adds a recurrent sequential perceiver adapter to a frozen CLIP ViT, enriches class prototypes with LLM-generated text descriptions, and matches support/query videos with unbalanced optimal transport.
Problem
Few-shot action recognition methods treat frames largely order-invariantly, so they fail to distinguish videos whose category depends on frame order (e.g., reversed clips).
Approach
A sequential perceiver adapter that uses a temporal token to recurrently attend to each frame in temporal order (frame-by-frame), parallel to frozen ViT blocks, combined with LLM-expanded textual class corpora and unbalanced optimal transport matching for FSAR.
Key results
- TSAM with CLIP ViT-B/16 reaches 60.5% 1-shot and 66.7% 5-shot on SSv2-Small, and 65.8% 1-shot on SSv2-Full, versus MA-CLIP's 59.1% and 63.3%.
- On HMDB-51 1-shot TSAM (ViT-B) reaches 84.5% versus 83.4% for MA-CLIP; on UCF101 1-shot 98.3% versus 97.0% for CLIP-FSAR.
- Under temporal reversal, TSAM drops 3.2% on SSv2-Full versus essentially no drop (62.1 to 62.0) for CLIP-FSAR.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use TSAM instead of MA-CLIP | CLIP-ViT-B few-shot action recognition · SSv2-Small 1-shot | accuracy: 59.1 -> 60.5 | Helped |
| Use TSAM instead of MA-CLIP | CLIP-ViT-B few-shot action recognition · SSv2-Small 5-shot | accuracy: 64.5 -> 66.7 | Helped |
| Use TSAM instead of MA-CLIP | CLIP-ViT-B few-shot action recognition · SSv2-Full 1-shot | accuracy: 63.3 -> 65.8 | Helped |
| Use TSAM instead of MA-CLIP | CLIP-ViT-B few-shot action recognition · SSv2-Full 5-shot | accuracy: 72.3 -> 74.6 | Helped |
| Use TSAM instead of MA-CLIP | CLIP-ViT-B few-shot action recognition · HMDB51 1-shot | accuracy: 83.4 -> 84.5 | Helped |
| Use TSAM instead of MA-CLIP | CLIP-ViT-B few-shot action recognition · HMDB51 5-shot | accuracy: 87.9 -> 88.9 | Helped |
| Use TSAM instead of CLIP-FSAR | CLIP-ViT-B few-shot action recognition · UCF101 5-shot | accuracy: 99.1 -> 99.3 | Helped |
| Use TSAM instead of MA-CLIP | CLIP-ViT-B few-shot action recognition · Kinetics 5-shot | accuracy: 96.0 -> 97.1 | Helped |
| Add Sequential Perceiver (ablation) | TSAM · SSv2-Small 1-shot | accuracy: 54.6 -> 58.2 | Helped |
| Add Sequential Perceiver (ablation) | TSAM · HMDB51 1-shot | accuracy: 77.1 -> 82.2 | Helped |
| Add Textual Corpus to Sequential Perceiver (ablation) | TSAM · SSv2-Small 1-shot | accuracy: 58.2 -> 59.0 | Helped |
| Add Textual Corpus to Sequential Perceiver (ablation) | TSAM · HMDB51 1-shot | accuracy: 82.2 -> 83.9 | Helped |
Surprising findings
- Competitors such as CLIP-FSAR show essentially no performance drop on HMDB-51 and Kinetics100 when frames are temporally reversed, suggesting they rely on static appearance cues.
Limitations
- Performance drop under temporal reversal is modest on HMDB-51 and Kinetics, so temporal order sensitivity is limited on scene-heavy datasets.
- Reliance on unnamed LLMs for textual corpus generation; the prompts and generated corpora are not detailed.
- Gains over strongest ViT baselines are small on some datasets (e.g., 0.2% on UCF101 5-shot).
Experimental setup
Datasets: HMDB-51 UCF101 Kinetics SSv2-Small SSv2-Full
Benchmarks: SSv2-Small SSv2-Full HMDB-51 UCF101 Kinetics
Metrics: 5-way 1-shot accuracy 5-way 5-shot accuracy
Baselines: CMN OTAM AmeFuNet TRX SPRN HyRSM T3A2N MoLo MGCSM SA-CT CLIP-FSAR MA-CLIP
Builds on: CLIP CLIP-FSAR Perceiver unbalanced optimal transport CLIP-Adapter
Closest papers by meaning
- Temporal Alignment-Free Video Matching for Few-shot Action Recognition (CVPR 2025 · similarity 0.89)
- Learning Causal Domain-Invariant Temporal Dynamics for Few-Shot Action Recognition (ICML 2024 · similarity 0.88)
- Enhancing Temporal Understanding in Video-LLMs through Stacked Temporal Attention in Vision Encoders (NeurIPS 2025 · similarity 0.87)
- SeFAR: Semi-supervised Fine-grained Action Recognition with Temporal Perturbation and Learning Stabilization (AAAI 2025 · similarity 0.86)
- TAMT: Temporal-Aware Model Tuning for Cross-Domain Few-Shot Action Recognition (CVPR 2025 · similarity 0.86)
- OST: Refining Text Knowledge with Optimal Spatio-Temporal Descriptor for General Video Recognition (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.