Research map / Vision-language and multimodal / Multimodal LLMs
SPORTU: A Comprehensive Sports Understanding Benchmark for Multimodal Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces SPORTU, a two-part sports understanding benchmark (900 text MCQs with human explanations and 12,048 video QA pairs over 1,701 slow-motion clips across 7 sports) and evaluates LLMs and MLLMs with several prompting strategies.
Problem
Existing sports QA datasets are either text-only without reasoning explanations or video-based but limited to one sport or lacking multi-level rule-based reasoning, so MLLMs' deep sports rule understanding is not comprehensively evaluated.
Approach
A multi-level sports benchmark pairing a pure-text QA set with human-written per-option explanations and a slow-motion multi-sport video QA set with easy/medium/hard tiers, including multi-angle clips and foul/rule-violation questions, evaluated under no-CoT, CoT, and answer-then-rationale prompting.
Key results
- On SPORTU-text, GPT-4o reaches 71.00% accuracy in the 5-shot CoT setting, below the expert level reported as over 90%.
- On SPORTU-video multiple-choice, Qwen2-VL-72B achieves the highest overall accuracy of 70.94% under X→Y, but only 44.12% on hard questions.
- Claude-3.5-Sonnet drops from 52.57% hard-level accuracy with direct answering to 39.32% when reasoning first (X→RY).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use five-shot CoT instead of zero-shot direct answering (ablation) | GPT-4o on SPORTU-text · SPORTU-text | accuracy: 70.22 -> 71.00 | Helped |
| Generate rationale before predicting the answer instead of answering directly (ablation) | Claude-3.5-Sonnet on SPORTU-video hard · SPORTU-video hard | accuracy: 53.06 -> 39.32 | Hurt |
| Generate rationale after predicting the answer instead of answering directly (ablation) | Claude-3.5-Sonnet on SPORTU-video · SPORTU-video multiple-choice | accuracy: 70.18 -> 69.52 | Hurt |
| Generate rationale before predicting the answer instead of answering directly (ablation) | GPT-4o on SPORTU-video · SPORTU-video multiple-choice | accuracy: 68.79 -> 65.56 | Hurt |
| Generate rationale before predicting the answer instead of answering directly (ablation) | MLLMs on SPORTU-video · SPORTU-video multiple-choice | accuracy | Mixed |
Surprising findings
- Models perform best when answering directly without a rationale; rationale-first (CoT) prompting lowers video QA accuracy for most models.
- On SPORTU-video open-ended tasks, no MLLM achieves an average G-Eval score above 3, and automatic metrics correlate weakly with human ratings (G-Eval Pearson 0.41).
Limitations
- Open-source video models were evaluated only at 7B scale (except Qwen2-VL-72B) due to compute limits.
- Automatic explanation metrics show weak correlation with human judgments, so open-ended evaluation is less reliable.
- Explanations were only annotated for a subset of the hardest video tasks due to annotation resources.
Experimental setup
Datasets: SportQA Sports-QA SoccerNet-XFoul
Benchmarks: SPORTU-text SPORTU-video
Metrics: accuracy ROUGE-L BERTScore BLEURT CTC G-Eval
Baselines: GPT-4o Claude-3.5-Sonnet Gemini 1.5 Pro Llama-3.1-405B Gemini 1.5 Flash GPT-4o-mini Claude-3.0-Haiku Qwen2-VL-72B LLaVA-NeXT VideoChat2
Builds on: SportQA Sports-QA SoccerNet-XFoul G-Eval chain-of-thought prompting
Closest papers by meaning
- SportR: A Benchmark for Multimodal Large Language Model Reasoning in Sports (ICLR 2026 · similarity 0.95)
- Towards Universal Soccer Video Understanding (CVPR 2025 · similarity 0.87)
- MVU-Eval: Towards Multi-Video Understanding Evaluation for Multimodal LLMs (NeurIPS 2025 · similarity 0.87)
- SoccerMaster: A Vision Foundation Model for Soccer Understanding (CVPR 2026 · similarity 0.87)
- BMMR: A Large-Scale Bilingual Multimodal Multi-Discipline Reasoning Dataset (NeurIPS 2025 · similarity 0.85)
- MME-SCI: A Comprehensive and Challenging Science Benchmark for Multimodal Large Language Models (AAAI 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.