Research map / Video / Video-language understanding
ExpertAF: Expert Actionable Feedback from Video
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Generates actionable coaching for physical skills from video (free-form expert commentary, expert demonstration retrieval, and expert 3D pose generation) by training a multimodal LLM on a weakly-supervised dataset mined from Ego-Exo4D expert commentary.
Problem
Existing skill-assessment from video only outputs scores or pairwise comparisons, leaving users to figure out how to correct mistakes; no method gives verbal and visual actionable coaching feedback on physical activities.
Approach
Unified autoregressive multimodal LLM that encodes learner video (InternVideo2 features), 3D pose tokens (PCT-based), and text, trained on pairs of incorrect learner and correct expert demonstrations mined from Ego-Exo4D commentary (LLM-summarized, body-region labeled, and temporally aligned via PA-MPJPE) to output commentary, retrieved expert demos, or generated expert poses.
Key results
- ExpertAF achieves BLEU-4 44.9 and ROUGE-L 54.6 for expert commentary, beating the best same-data baseline LLaVA-FT w/ pose (43.6 and 51.7).
- Expert demonstration retrieval median rank 158 vs 172 for the best baseline, with recall@50 of 19.1 vs 18.0.
- Human raters score ExpertAF up to 0.7 Likert points higher and up to 3x better than off-the-shelf video models.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ExpertAF instead of LLaVA-FT w/ pose | Expert commentary generation · Expert commentary generation | BLEU-4: 43.6 -> 44.9 | Helped |
| Use ExpertAF instead of LLaVA-FT w/ pose | Expert commentary generation · Expert commentary generation | ROUGE-L F1: 51.7 -> 54.6 | Helped |
| Use ExpertAF instead of LLaVA-FT w/ pose | Expert demonstration retrieval · Expert demonstration retrieval | recall@50: 18.0 -> 19.1 | Helped |
| Use ExpertAF instead of LLaVA-FT w/ pose | Expert demonstration retrieval · Expert demonstration retrieval | median rank: 172 -> 158 | Helped |
| Use ExpertAF instead of LLaVA-FT w/ pose | Expert pose generation · Expert pose generation | PA-MPJPE: 150 -> 135 | Helped |
| Remove video input (ablation) | ExpertAF · Expert commentary generation | BLEU-4: 44.9 -> 44.6 | Hurt |
| Remove pose input (ablation) | ExpertAF · Expert commentary generation | BLEU-4: 44.9 -> 44.2 | Hurt |
| Remove temporal alignment (ablation) | ExpertAF · Expert commentary generation | BLEU-4: 44.9 -> 42.0 | Hurt |
| Use one global correct pose per activity instead of the closest correct pose sequence (ablation) | ExpertAF · Expert demonstration retrieval | recall@50: 19.1 -> 16.5 | Hurt |
| Use the expert demonstration as privileged input at inference (ablation) | ExpertAF · Expert demonstration retrieval | recall@50: 19.1 -> 22.5 | Helped |
Surprising findings
- Video-captioning models (VideoChat2, LLaVA) produce verbose activity descriptions rather than critiques, so off-the-shelf captioners fail at feedback.
- PoseScript-style pose descriptions (e.g., 'head is turned to the left') are inadequate for expert feedback.
Limitations
- Pose sequences in Ego-Exo4D are often auto-generated and noisy.
- Struggles with feedback not directly visible, such as fatigue in climbing.
- Expert video generation is left as future work; only retrieval and pose generation are supported.
Experimental setup
Datasets: Ego-Exo4D
Benchmarks: Ego-Exo4D
Metrics: BLEU-4 METEOR ROUGE-L recall@50 median rank PA-MPJPE human Likert rating (1-4)
Baselines: InternVideo2-NN InternVideo2-FT VideoChat2 LLaVA LLaVA-FT LLaVA-FT w/ pose PoseScript PoseFix
Builds on: LLaVA InternVideo2 PCT Ego-Exo4D PoseScript
Closest papers by meaning
- Learning Skill-Attributes for Transferable Assessment in Video (NeurIPS 2025 · similarity 0.91)
- ExAct: A Video-Language Benchmark for Expert Action Analysis (NeurIPS 2025 · similarity 0.88)
- Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectives (CVPR 2024 · similarity 0.86)
- From 3D Pose to Prose: Biomechanics-Grounded Vision-Language Coaching (CVPR 2026 · similarity 0.86)
- LucidAction: A Hierarchical and Multi-model Dataset for Comprehensive Action Quality Assessment (NeurIPS 2024 · similarity 0.85)
- ActionAtlas: A VideoQA Benchmark for Domain-specialized Action Recognition (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.