Research map / Video / Video-language understanding
SportsHHI: A Dataset for Human-Human Interaction Detection in Sports Videos
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces the video human-human interaction detection task and the SportsHHI dataset (34 high-level interaction classes from basketball and volleyball), with a two-stage Faster-RCNN-style baseline using motion, context, and relative-position features plus self-attention.
Problem
Existing video visual relation detection datasets rarely include human-human interactions in multi-person scenes and use low-level, appearance-recognizable relation types, so they do not test spatio-temporal context reasoning about high-level interactions.
Approach
Define a human-human interaction triplet task and annotate high-level, tactical/technical/confrontational interactions among athletes in team sports videos at 5FPS keyframes, then benchmark with a two-stage pipeline that enumerates person pairs as proposals and classifies them using motion features, union-box context, spatial masks, and inter-proposal self-attention.
Key results
- The proposed two-stage baseline with SlowFast-R50 reaches 7.52 mAP and 85.78 R@150 in HHICls mode, outperforming STTran, HORT, SlowFast and ACARN on SportsHHI.
- With a VideoMAE ViT-B backbone, the baseline reaches 10.69 mAP in HHICls and 4.93 mAP in HHIDet mode.
- Human box detection achieves 98.6 recall at IoU 0.5 but interaction proposal recall drops to 37.7 at IoU 0.9.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ViT-B instead of SlowFast-R50 as the baseline backbone | SportsHHI baseline · SportsHHI validation HHICls | mAP: 7.52 -> 10.69 | Helped |
| Use ViT-B instead of SlowFast-R50 as the baseline backbone | SportsHHI baseline · SportsHHI validation HHIDet | mAP: 3.36 -> 4.93 | Helped |
| Add subject/object motion features to a context-only representation (ablation) | SportsHHI baseline · SportsHHI validation HHICls | mAP: 2.07 -> 5.19 | Helped |
| Add relative position encoding to visual motion and context features (ablation) | SportsHHI baseline · SportsHHI validation HHICls | mAP: 5.43 -> 7.52 | Helped |
| Add information exchange among proposals using multi-head self-attention (ablation) | SportsHHI baseline · SportsHHI validation HHICls | mAP: 5.43 -> 7.52 | Helped |
| Set positive:negative proposal ratio to 1:10 instead of 1:1 (ablation) | SportsHHI baseline · SportsHHI validation HHICls | mAP: 6.97 -> 7.52 | Helped |
| Set positive:negative proposal ratio to 1:15 or 1:25 instead of 1:10 (ablation) | SportsHHI baseline · SportsHHI validation HHICls | mAP: 7.52 -> 7.02; 7.52 -> 6.89 | Hurt |
| Use separate subject and object spatial masks instead of a single mask (ablation) | SportsHHI baseline · SportsHHI validation HHICls | mAP: 6.77 -> 7.52 | Helped |
| Use motion features only rather than the full feature representation (ablation) | SportsHHI baseline · SportsHHI validation HHICls | mAP: 7.52 -> 5.00 | Hurt |
| Use context features only rather than the full feature representation (ablation) | SportsHHI baseline · SportsHHI validation HHICls | mAP: 7.52 -> 2.07 | Hurt |
| Use relative position encoding only rather than the full feature representation (ablation) | SportsHHI baseline · SportsHHI validation HHICls | mAP: 7.52 -> 1.13 | Hurt |
| Add context features to motion and position features (ablation) | SportsHHI baseline · SportsHHI validation HHICls | mAP: 6.15 -> 7.52 | Helped |
Surprising findings
- Existing video scene graph generation methods (STTran, HORT) underperform action detectors because appearance-based relation features and object-category priors do not transfer when all entities are humans.
Limitations
- Covers only basketball and volleyball with 34 classes, and the long-tailed class distribution leaves rare classes with few instances.
- Many false positives come from distinguishing non-interacting people in crowded scenes, and the baseline's absolute performance remains low (mAP around 7.5 to 10.7).
Experimental setup
Datasets: SportsHHI MultiSports
Benchmarks: SportsHHI
Metrics: mAP Recall@K
Baselines: STTran HORT SlowFast ACARN ViT-B (VideoMAE)
Builds on: VidVRD STTran HORT SlowFast Faster R-CNN
Closest papers by meaning
- Learning Human-Object Interaction as Groups (NeurIPS 2025 · similarity 0.87)
- HIG: Hierarchical Interlacement Graph Approach to Scene Graph Generation in Video Understanding (CVPR 2024 · similarity 0.86)
- FineSports: A Multi-person Hierarchical Sports Video Dataset for Fine-grained Action Understanding (CVPR 2024 · similarity 0.86)
- Towards Universal Soccer Video Understanding (CVPR 2025 · similarity 0.86)
- ContextHOI: Spatial Context Learning for Human-Object Interaction Detection (AAAI 2025 · similarity 0.85)
- Discovering Syntactic Interaction Clues for Human-Object Interaction Detection (CVPR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.