Research map / Vision-language and multimodal / Cross-modal learning
Language-Guided Audio-Visual Learning for Long-Term Sports Assessment
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MLAVL, a language-guided audio-visual framework that uses multidimensional action-prompt knowledge graphs to model audio-action-visual correlations for long-term sports assessment with low parameter cost.
Problem
Long-term sports assessment requires judging movement quality and action-music coordination, but background music is weakly related to actions and prior audio-visual models rely on large parameter counts to learn these associations.
Approach
Builds action knowledge graphs from text prompts of basic sports actions (visual "a video of [action]" and audio "a music suitable for [action]") that guide audio and visual temporal graphs via GCNs; adds a shared-specific context encoder, an audio-visual cross-modal fusion module (global + clip-wise), and a dual-branch prompt-guided grading module.
Key results
- On FS1000, MLAVL reaches Avg. Spearman 0.90 and Avg. MSE 10.39, outperforming PAMFN (0.87 / 16.80).
- On Fis-V, MLAVL achieves Avg. Sp. Corr. 0.823 and Avg. MSE 13.31 with 3.82M parameters and 0.778 GFLOPs.
- On Rhythmic Gymnastics, MLAVL reaches Avg. Spearman 0.849 and Avg. MSE 4.47, improving MSE by 1.06 over GDLT; plugging MAG² into nine visual-only LOGO methods improves their Spearman correlation on average.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add S²CE (ablation) | MLAVL baseline · FS1000 (TES) | Spearman correlation: 0.835 -> 0.848 | Helped |
| Add MAG² (ablation) | MLAVL · FS1000 (TES) | Spearman correlation: 0.848 -> 0.876 | Helped |
| Add AVCF (ablation) | MLAVL without DPG · FS1000 (TES) | Spearman correlation: 0.876 -> 0.887 | Helped |
| Add DPG (ablation) | MLAVL with S²CE, MAG², and AVCF · FS1000 (TES) | Spearman correlation: 0.887 -> 0.917 | Helped |
| Add S²CE (ablation) | MLAVL · FS1000 (TES) | Spearman correlation: 0.891 -> 0.917 | Helped |
| Add MAG² (ablation) | MLAVL · FS1000 (TES) | Spearman correlation: 0.879 -> 0.917 | Helped |
| Add AVCF (ablation) | MLAVL · FS1000 (TES) | Spearman correlation: 0.894 -> 0.917 | Helped |
| Add triplet loss LT L (ablation) | MLAVL · FS1000 (TES) | Spearman correlation: 0.886 -> 0.917 | Helped |
| Add cross-entropy loss LCE (ablation) | MLAVL · FS1000 (TES) | Spearman correlation: 0.894 -> 0.917 | Helped |
| Add triplet loss LT L and cross-entropy loss LCE (ablation) | MLAVL · FS1000 (TES) | Spearman correlation: 0.875 -> 0.917 | Helped |
| Add MAG² action-visual graph guidance | Visual-only methods · LOGO | Spearman correlation: 0.542 -> 0.582; 0.762 -> 0.804; 0.647 -> 0.654; 0.697 -> 0.723; 0.589 -> 0.621; 0.541 -> 0.640; 0.681 -> 0.699; 0.661 -> | Helped |
Surprising findings
- Existing SOTA audio-visual models gain performance mainly by adding parameters while the audio-action relationship is weakly exploited, per the authors' motivation.
- Predicted scores in prior QTD work were not constrained to the label range, inflating MSE; the authors note this issue in their comparisons.
Limitations
- Effectiveness relies on rule-based action-music consistency requirements in long-term sports; may not transfer to other settings.
- Performance may suffer in specialized scenes where audio contains sparse useful or irrelevant information.
Experimental setup
Datasets: FS1000 Fis-V Rhythmic Gymnastics LOGO
Benchmarks: FS1000 Fis-V Rhythmic Gymnastics LOGO
Metrics: Spearman's rank correlation mean square error relative L2-distance
Baselines: C3D-LSTM MSCADC MS-LSTM CoRe GDLT TPT T2CR CoFInAl QTD M-BERT MLP-Mixer SGN
Builds on: ViFi-CLIP Video Swin Transformer Audio Spectrogram Transformer MLP-Mixer GDLT
Closest papers by meaning
- JointAVBench: A Benchmark for Joint Audio-Visual Reasoning Evaluation (ICLR 2026 · similarity 0.86)
- MotionBench: Benchmarking and Improving Fine-grained Video Motion Understanding for Vision Language Models (CVPR 2025 · similarity 0.86)
- Towards Universal Soccer Video Understanding (CVPR 2025 · similarity 0.86)
- Unleashing Hour-Scale Video Training for Long Video-Language Understanding (NeurIPS 2025 · similarity 0.85)
- From Senses to Decisions: The Information Flow of Auditory and Visual Perception in Multimodal LLMs (NeurIPS 2026 · similarity 0.85)
- Adapting to the Unknown: Training-Free Audio-Visual Event Perception with Dynamic Thresholds (CVPR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.