Research map / Video / Action recognition
ActionAtlas: A VideoQA Benchmark for Domain-specialized Action Recognition
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces ActionAtlas v1.0, a multiple-choice VideoQA benchmark of 934 short sports videos testing fine-grained, domain-specialized action recognition, and evaluates open, proprietary VLMs and non-expert humans.
Problem
Existing video QA and action recognition benchmarks mostly cover simplistic actions identifiable from a single frame, so they do not test whether VLMs can distinguish subtle, similar-looking expert-level sports moves across temporal context.
Approach
A scalable collection pipeline in which GPT-4 generates action lists for sports (expansion and squeeze prompting), YouTube metadata and Whisper transcripts with GPT-4 narrow candidate videos, and crowd workers and authors verify and localize actions; questions refer to actors by natural-language attributes and use LLM-written hard negatives.
Key results
- GPT-4o reaches up to 42.95% accuracy with 16 frames (abstract reports 45.52%), versus 20.91% random chance.
- Non-expert crowd workers given action descriptions achieve 61.64% accuracy.
- Qwen2-VL-7B is the best open model at 30.24% with 16 frames; most other open models perform near chance.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase sampled frames from 2 to 16 (ablation) | Qwen2-VL-7B · ActionAtlas | accuracy: 24.07 -> 30.24 | Helped |
| Increase sampled frames from 16 to 32 (ablation) | Qwen2-VL-7B · ActionAtlas | accuracy: 30.24 -> 29.33 | Hurt |
| Increase sampled frames from 1 to 16 (ablation) | GPT-4o · ActionAtlas | accuracy: 33.08 -> 42.95 | Helped |
| Increase sampled frames (ablation) | GPT-4 Turbo and GPT-4o-mini · ActionAtlas | accuracy | No clear effect |
| Add action choice descriptions (ablation) | GPT-4o · ActionAtlas | accuracy: 42.95 -> 44.05 | No clear effect |
| Add action choice descriptions (ablation) | Qwen2-VL-7B · ActionAtlas | accuracy: 30.24 -> 33.13 | No clear effect |
| Add action choice descriptions (ablation) | Video-LLaMA · ActionAtlas | accuracy: 22.71 -> 21.85 | Hurt |
| Add action choice descriptions (ablation) | LLaVA-Next-Video · ActionAtlas | accuracy: 20.77 -> 22.20 | No clear effect |
| Add action choice descriptions (ablation) | Gemini 1.5 Pro · ActionAtlas | accuracy: 32.37 -> 37.00 | No clear effect |
| Use chain-of-thought reasoning without choice descriptions (ablation) | Evaluated VLMs · ActionAtlas | accuracy | Hurt |
| Reason step by step across consecutive frames (ablation) | Evaluated VLMs · ActionAtlas | accuracy | No clear effect |
| Use all video frames instead of default 1 fps (ablation) | Gemini 1.5 Pro in video mode · ActionAtlas | accuracy: 32.37 -> 35.59 | No clear effect |
Surprising findings
- Gemini 1.5 Pro in default 1 fps video mode does not improve when all frames are used (35.59% vs 32.37%), and sampling Gemini processes only the middle frame each second.
- Adding action choice descriptions did not significantly improve most models, and chain-of-thought prompting significantly reduced performance for all models.
Limitations
- Limited to the sports domain and small in size (934 videos).
- No action taxonomy; action list may be incomplete.
- Gemini refusals, especially on wrestling content, affect evaluation results.
Experimental setup
Datasets: YouTube videos (crawled metadata, about 4.5M)
Benchmarks: ActionAtlas v1.0
Metrics: top-1 accuracy bootstrap 95% confidence intervals inference TFLOPs input video tokens
Baselines: CLIP ViT-L-14-336 mPLUG-Owl Video-LLaMA Video-LaVIT VideoChat2 LLaVA-Next-Video Qwen2-VL-7B Gemini 1.5 Flash Gemini 1.5 Pro GPT-4 Turbo GPT-4o-mini GPT-4o
Builds on: Whisper CLIP Elasticsearch
Closest papers by meaning
- LLaVAction: evaluating and training multi-modal large language models for action understanding (ICLR 2026 · similarity 0.87)
- Unsupervised Action Segmentation via Fast Learning of Semantically Consistent Actoms (AAAI 2024 · similarity 0.87)
- FineParser: A Fine-grained Spatio-temporal Action Parser for Human-centric Action Quality Assessment (CVPR 2024 · similarity 0.87)
- SeFAR: Semi-supervised Fine-grained Action Recognition with Temporal Perturbation and Learning Stabilization (AAAI 2025 · similarity 0.87)
- Action Detail Matters: Refining Video Recognition with Local Action Queries (CVPR 2025 · similarity 0.87)
- FineSports: A Multi-person Hierarchical Sports Video Dataset for Fine-grained Action Understanding (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.