Research map / Video / Video-language understanding
Learning Skill-Attributes for Transferable Assessment in Video
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Learns transferable skill-attributes from expert sports commentary and trains a multimodal LLM (Llama-3.1-8B with LoRA) to generate skill-attributes, actionable feedback, and proficiency for in-domain and cross-sport zero-shot skill assessment.
Problem
Video skill assessment currently works only within individual sports with expert supervision; it needs to transfer to long-tail sports with little labeled data, both to name incorrectly performed skill-attributes and to give actionable feedback and proficiency.
Approach
Extract sport-agnostic skill-attributes (e.g., balance, control, hand positioning) from expert commentary with an LLM, pretrain a multimodal LLM to generate them from video, then condition feedback and proficiency generation on the predicted attributes so the representation transfers across sports.
Key results
- Skill-attribute generation IoU@0.7 on Ego-Exo4D rises to 25.7 for CrossTrainer vs 15.0 for the best baseline ExpertAF; on QEVD 37.6 vs 28.1.
- Actionable feedback on Ego-Exo4D reaches METEOR 51.7 and ROUGE-L 57.8 vs 49.6 and 54.6 for ExpertAF; zero-shot performance drops at most 4% vs 17% for the baseline.
- Proficiency accuracy on Ego-Exo4D improves over EgoVLPv2 by up to about 6 points (e.g., soccer 68.8 vs 62.5).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add skill-attribute-guided training instead of direct end-to-end training (ablation) | CrossTrainer · Ego-Exo4D actionable feedback | BLEU@4: 43.8 -> 45.6 | Helped |
| add skill-attribute-guided training instead of direct end-to-end training (ablation) | CrossTrainer · Ego-Exo4D actionable feedback | METEOR: 48.8 -> 51.7 | Helped |
| add skill-attribute-guided training instead of direct end-to-end training (ablation) | CrossTrainer · Ego-Exo4D actionable feedback | ROUGE-L: 52.3 -> 57.8 | Helped |
| add skill-attribute-guided training instead of direct end-to-end training (ablation) | CrossTrainer · QEVD actionable feedback | METEOR: 12.1 -> 17.6 | Helped |
| add skill-attribute-guided training instead of direct end-to-end training (ablation) | CrossTrainer · QEVD actionable feedback | ROUGE-L: 10.8 -> 18.1 | Helped |
| add skill-attribute-guided training instead of direct end-to-end training (ablation) | CrossTrainer · QEVD actionable feedback | BERT score: 86.0 -> 87.8 | Helped |
| use generated skill-attributes rather than attribute retrieval | CrossTrainer · Ego-Exo4D | IoU@0.7: 19.7 -> 25.7 | Helped |
| use generated skill-attributes rather than attribute retrieval | CrossTrainer · QEVD | IoU@0.7: 32.4 -> 37.6 | Helped |
| inject 10% random skill-attribute noise (ablation) | CrossTrainer · Ego-Exo4D actionable feedback | ROUGE-L: 57.8 -> 56.2 | Hurt |
| inject 20% random skill-attribute noise (ablation) | CrossTrainer · Ego-Exo4D actionable feedback | ROUGE-L: 57.8 -> 54.9 | Hurt |
| inject 30% random skill-attribute noise (ablation) | CrossTrainer · Ego-Exo4D actionable feedback | ROUGE-L: 57.8 -> 53.1 | Hurt |
| inject 50% random skill-attribute noise (ablation) | CrossTrainer · Ego-Exo4D actionable feedback | ROUGE-L: 57.8 -> 50.2 | Hurt |
Surprising findings
- Human raters judged only about 75% of generated feedback on YouTube long-tail videos as actionable and correct.
- Feedback is hard to ground for non-visual aspects such as intent and decision.
Limitations
- Struggles with feedback about aspects not visible in video, such as intent.
- Commentary is subjective and multiple valid feedback statements can exist, complicating evaluation.
- Does not model terrain, opponents, or multi-player team interactions.
Experimental setup
Datasets: Ego-Exo4D QEVD
Benchmarks: Ego-Exo4D QEVD YouTube in-the-wild sports videos
Metrics: IoU@0.7 IoU@0.8 IoU@1.0 BLEU-4 METEOR ROUGE-L BERT score classification accuracy
Baselines: InternVideo2-NN InternVideo2-FT VideoChat2 LLaVA LLaVA-FT Stream-VLM ExpertAF Attribute-Retrieval Video-ChatGPT LLaMA-VID PoseScript/Fix EgoVLPv2
Builds on: ExpertAF LLaVA Stream-VLM EgoVLPv2
Closest papers by meaning
- ExpertAF: Expert Actionable Feedback from Video (CVPR 2025 · similarity 0.91)
- LucidAction: A Hierarchical and Multi-model Dataset for Comprehensive Action Quality Assessment (NeurIPS 2024 · similarity 0.86)
- SportR: A Benchmark for Multimodal Large Language Model Reasoning in Sports (ICLR 2026 · similarity 0.86)
- ViSTec: Video Modeling for Sports Technique Recognition and Tactical Analysis (AAAI 2024 · similarity 0.86)
- ExAct: A Video-Language Benchmark for Expert Action Analysis (NeurIPS 2025 · similarity 0.86)
- Language-Guided Audio-Visual Learning for Long-Term Sports Assessment (CVPR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.