Research map / Vision-language and multimodal / Multimodal LLMs
FSBench: A Figure Skating Benchmark for Advancing Artistic Sports Understanding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces FSAnno, a multi-level figure skating dataset, and FSBench, a benchmark of prior-knowledge, action-level, and performance-level tasks, and fine-tunes MotionGPT into SkateLLM for motion captioning.
Problem
Existing figure skating datasets focus on single tasks (action recognition, segmentation, or scoring) and lack comprehensive technical and artistic annotations, while sports QA benchmarks focus on ball sports and do not evaluate artistry or performance quality.
Approach
A multi-level, multi-modal figure skating dataset (motion, skeleton, text, audio-derived commentary, GOE/PCS scores) with a benchmark spanning prior knowledge, individual actions, and entire performances, plus AutoDQ-based evaluation of generated descriptions and an instruction-tuned motion LLM (SkateLLM).
Key results
- On the FSBench prior-knowledge test, GPT-4 scored 73.0% (Event Info), 78.0% (Rules), and 87.9% (Quiz), versus 59.0/64.8/72.7 for GPT-3.5-turbo and 47.0/51.4/63.6 for LLaVA 13B.
- On AutoDQ motion captioning, SkateLLM reached F1 38.0 versus 7.1 for Motion-GPT, with recall 58.333 vs 3.772 and precision 61.622 vs 27.456.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Instruction-tune Motion-GPT on figure skating descriptions using FSAnno data | Motion-GPT, producing SkateLLM · FSBench figure skating element description | F1: 7.1 -> 38.0 | Helped |
| Instruction-tune Motion-GPT on figure skating descriptions using FSAnno data | Motion-GPT, producing SkateLLM · FSBench figure skating element description | Recall: 3.772 -> 58.333 | Helped |
| Instruction-tune Motion-GPT on figure skating descriptions using FSAnno data | Motion-GPT, producing SkateLLM · FSBench figure skating element description | Precision: 27.456 -> 61.622 | Helped |
Surprising findings
- Only GPT-4 reached the 'All-Star Striker' level on an external figure skating quiz; other LLMs performed average, suggesting weak artistic-sport knowledge.
Limitations
- SkateLLM has lower recall than precision, struggling with rotation counts and fine-grained distinctions.
- Evaluation of video-based and most motion-based LLMs is only partial in the main text; full results deferred to supplementary material.
Experimental setup
Datasets: FSAnno general motion understanding datasets (unnamed)
Benchmarks: FSBench
Metrics: accuracy AutoDQ precision AutoDQ recall AutoDQ F1
Baselines: GPT-3.5-turbo GPT-4 LLaVA 13B Motion-GPT
Builds on: MotionGPT AutoDQ 4DHumans HRNet Whisper
Closest papers by meaning
- SportR: A Benchmark for Multimodal Large Language Model Reasoning in Sports (ICLR 2026 · similarity 0.84)
- LucidAction: A Hierarchical and Multi-model Dataset for Comprehensive Action Quality Assessment (NeurIPS 2024 · similarity 0.82)
- ViSTec: Video Modeling for Sports Technique Recognition and Tactical Analysis (AAAI 2024 · similarity 0.81)
- Seeing Beyond the Crop: Using Language Priors for Out-of-Bounding Box Keypoint Prediction (NeurIPS 2024 · similarity 0.81)
- SPORTU: A Comprehensive Sports Understanding Benchmark for Multimodal Large Language Models (ICLR 2025 · similarity 0.81)
- ExAct: A Video-Language Benchmark for Expert Action Analysis (NeurIPS 2025 · similarity 0.80)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.