Research map / Video / Action recognition
Storyboard-guided Alignment for Fine-grained Video Action Recognition
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Decomposes global action class names into LLM-generated atomic sub-texts, filters them with a text prompt perplexity metric, and uses them to compute coarse and fine-grained frame-weighted video embeddings for CLIP-based video-text matching.
Problem
Fine-grained video action recognition is framed as video-text matching, but aligning frames to global class-name text causes mismatches because videos with different global semantics share atomic actions and atomic actions are unevenly distributed and sometimes unrelated to the global label.
Approach
Generate atomic sub-texts from class names with an LLM (storyboard-style decomposition), select them with a text prompt perplexity score, and use the augmented global text to weight frames for a coarse embedding and sub-texts to weight frames for a fine-grained embedding, then fuse both for CLIP-based classification.
Key results
- Achieves 84.5% top-1 on Kinetics-400 with an 8-frame single-view ViT-B/16, surpassing BIKE (83.9% with 4x3 views).
- Reaches 51.1 mAP on Charades, 0.7 above BIKE (50.4).
- Zero-shot results of 78.9% on UCF-101 and 56.6% on HMDB-51; two-shot results of 66.0 on HMDB-51, 92.0 on UCF-101, 73.8 on Kinetics-400.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Storyboard-guided Alignment instead of ILA ViT-B/16 | Video action recognition on Kinetics-400 · Kinetics-400 | Top-1 accuracy (%): 84.0 -> 84.5 | Helped |
| Use Storyboard-guided Alignment instead of ILA ViT-B/16 | Video action recognition on Kinetics-400 · Kinetics-400 | Top-5 accuracy (%): 96.6 -> 96.7 | Helped |
| Use Storyboard-guided Alignment instead of BIKE | Video action recognition · Charades | mAP: 50.4 -> 51.1 | Helped |
| Use Storyboard-guided Alignment instead of M2-CLIP | Zero-shot action recognition · UCF-101 | Accuracy: 78.7 -> 78.9 | Helped |
| Use Storyboard-guided Alignment instead of BIKE | Zero-shot action recognition · HMDB-51 | Accuracy: 55.6 -> 56.6 | Helped |
| Use Storyboard-guided Alignment instead of BIKE | 2-shot action recognition · HMDB-51 | Accuracy: 65.0 -> 66.0 | Helped |
| Use Storyboard-guided Alignment instead of BIKE | 2-shot action recognition · UCF-101 | Accuracy: 91.3 -> 92.0 | Helped |
| Use Storyboard-guided Alignment instead of BIKE | 2-shot action recognition · Kinetics-400 | Accuracy: 72.9 -> 73.8 | Helped |
| Add coarse video embedding to fine-grained-only model (ablation) | Storyboard-guided Alignment · Kinetics-400 | Top-1 accuracy (%): 82.7 -> 84.5 | Helped |
| Add fine-grained video embedding to coarse-only model (ablation) | Storyboard-guided Alignment · Kinetics-400 | Top-1 accuracy (%): 83.1 -> 84.5 | Helped |
| Add coarse and fine-grained components (ablation) | VideoPrompt · Kinetics-400 | Top-1 accuracy (%): 76.9 -> 79.3 | Helped |
| Add coarse and fine-grained components (ablation) | ATM · Kinetics-400 | Top-1 accuracy (%): 82.8 -> 82.9 | Helped |
Limitations
- Sub-text set and code are only promised to be released ('will be made available online'); no link given.
- Relies on the quality of LLM-generated atomic action descriptions, with selection by a heuristic perplexity metric.
- Reported correlation of TPP with performance is based on only 11 sub-text groups.
Experimental setup
Datasets: Kinetics-400 Charades UCF-101 HMDB-51
Benchmarks: Kinetics-400 Charades UCF-101 HMDB-51
Metrics: top-1 accuracy top-5 accuracy mAP
Baselines: ActionCLIP ViViT ST-Adapter EVL MTV VideoSwin ATM DIST AIM ILA X-CLIP VideoPrompt
Builds on: CLIP BIKE ActionCLIP InfoNCE
Closest papers by meaning
- SeFAR: Semi-supervised Fine-grained Action Recognition with Temporal Perturbation and Learning Stabilization (AAAI 2025 · similarity 0.89)
- Temporal Alignment-Free Video Matching for Few-shot Action Recognition (CVPR 2025 · similarity 0.89)
- Align Before Adapt: Leveraging Entity-to-Region Alignments for Generalizable Video Action Recognition (CVPR 2024 · similarity 0.88)
- OST: Refining Text Knowledge with Optimal Spatio-Temporal Descriptor for General Video Recognition (CVPR 2024 · similarity 0.87)
- ActionAtlas: A VideoQA Benchmark for Domain-specialized Action Recognition (NeurIPS 2024 · similarity 0.86)
- VideoComp: Advancing Fine-Grained Compositional and Temporal Alignment in Video-Text Models (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.