Research map / Video / Video-language understanding

$F^3Set$: Towards Analyzing Fast, Frequent, and Fine-grained Events from Videos

ICLR 2025 · Video › Video-language understanding · approach: Event and anomaly reasoning in video

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces F3Set, a benchmark of fast, frequent, fine-grained sports event datasets (mainly tennis, ~1,108 event types), evaluates temporal action understanding methods on it, and proposes F3ED, an end-to-end TSM-based detector with multi-label classification and a BiGRU contextual refinement module.

Problem

Detecting sequences of fast, frequent, fine-grained events with precise timestamps (e.g., tennis shots) is poorly served by existing temporal action localization/segmentation methods and datasets, and MLLMs like GPT-4 perform poorly on it.

Approach

Treat each event as a multi-label vector of sub-class elements rather than a single multi-class label, use dense frame-wise end-to-end training with a TSM encoder, and refine predicted event sequences with a bidirectional GRU contextual module that learns event-wise causal constraints.

Key results

• On F3Set-tennis (G_high), F3ED with TSM encoder reaches F1_evt 40.3, F1_elm 75.2, Edit 74.0, versus best baseline TSM+E2E-Spot at 31.4 / 71.4 / 68.7.
• F3ED improves over the best baseline on the semi-F3 datasets, e.g., FineGym F1_evt 70.9 vs 62.1 for E2E-Spot and CCTV-Pipe F1_evt 37.0 vs 31.3 for G-TAD.
• Performance drops as granularity increases; multi-label classification alone raises G_high F1_evt from 31.4 to 37.9 and adding BiGRU CTX gives 40.3.

Measured effects reported in the paper

Surprising findings

• GPT-4V performed poorly on fine-grained tennis rally understanding (incorrect shot types and directions), so it was excluded from comparisons.
• Larger video encoders (I3D, VTN, SlowFast) underperformed simpler TSM with RegNet-Y 200MF, suggesting fine-grained short-span temporal cues matter more than encoder complexity.

Limitations

• Tennis case study is the main domain; other domains (badminton, table tennis, doubles) are described with less detail and fewer results.
• Dataset relies on public broadcast videos, providing only YouTube links, and annotations were produced by a small team (8 annotators) with majority voting.
• Event timestamps are defined as instantaneous ball-racket contact, which may not generalize to other event types.

Experimental setup

Datasets: F3Set-tennis F3Set-badminton F3Set-table tennis F3Set-tennis doubles ShuttleSet FineDiving FineGym SoccerNetV2 CCTV-Pipe ImageNet-1K

Benchmarks: F3Set-tennis ShuttleSet FineDiving FineGym SoccerNetV2 CCTV-Pipe

Metrics: F1_evt F1_elm Edit score mean F1 with temporal tolerance

Baselines: TSN SlowFast I3D VTN TSM MS-TCN ASFormer G-TAD ActionFormer E2E-Spot ST-GCN++ PoseConv3D

Builds on: E2E-Spot TSM ActionFormer MS-TCN TSN

Closest papers by meaning

• Improving LLM Video Understanding with 16 Frames Per Second (ICML 2025 · similarity 0.84)
• Benchmarking the Robustness of Temporal Action Detection Models Against Temporal Corruptions (CVPR 2024 · similarity 0.84)
• Precise Event Spotting in Sports Videos: Solving Long-Range Dependency and Class Imbalance (CVPR 2025 · similarity 0.83)
• FineSports: A Multi-person Hierarchical Sports Video Dataset for Fine-grained Action Understanding (CVPR 2024 · similarity 0.83)
• FineParser: A Fine-grained Spatio-temporal Action Parser for Human-centric Action Quality Assessment (CVPR 2024 · similarity 0.83)
• MomentSeeker: A Task-Oriented Benchmark For Long-Video Moment Retrieval (NeurIPS 2025 · similarity 0.83)