Research map / Vision-language and multimodal / Human and embodied modeling
What to Say and When to Say it: Live Fitness Coaching as a Testbed for Situated Interaction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces QEVD, a large fitness-coaching video dataset and live-feedback benchmark (QEVD-FIT-COACH), and a streaming video-language baseline STREAM-VLM that learns when and what to say via <next>/<feedback> action tokens.
Problem
Existing vision-language models are turn-based and cannot proactively give timely, situated feedback on live user activity; there are no datasets for corrective real-time fitness coaching.
Approach
A causal 3D-CNN motion backbone fused via cross-attention into a LLaMA-2-7B LM, with special <next> and <feedback> action tokens so the model decides on-the-fly when to observe or speak in a video stream, trained end-to-end on coaching annotations.
Key results
- STREAM-VLM achieves temporal F-score 0.56 (vs 0.50 for turn-based baselines) and METEOR 0.127 on QEVD-FIT-COACH, the best among compared models.
- Zero-shot open VLMs score weakly, with LLM-Accuracy between 1.29 and 2.27 out of 5 and METEOR at most 0.104.
- Removing the 3D CNN backbone drops METEOR from 0.127 to 0.090 and LLM-Accuracy from 2.45 to 2.11.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add the 3D CNN vision backbone instead of a CLIP-based encoder (ablation) | STREAM-VLM · QEVD-FIT-COACH | METEOR: 0.090 -> 0.127 | Helped |
| add QEVD-FIT-300K short-clip pre-training (ablation) | STREAM-VLM · QEVD-FIT-COACH | METEOR: 0.095 -> 0.127 | Helped |
| add the <next> and <feedback> action tokens for interactive feedback (ablation) | STREAM-VLM · QEVD-FIT-COACH | T-F-Score: 0.50 -> 0.56 | Helped |
| fine-tune Video-ChatGPT on QEVD | Video-ChatGPT · QEVD-FIT-COACH | METEOR: 0.098 -> 0.108 | Helped |
Surprising findings
- A text-only Socratic LLaMA-2-7B fed 3D-CNN activity descriptions outperforms zero-shot state-of-the-art video-language models.
- Removing the action tokens yields similar fluency scores (METEOR 0.125) but a lower temporal F-score, showing the tokens mainly help timing.
Limitations
- Only one restricted domain (fitness) with domain-specific training data; broader domains and less data are open.
- Does not include speech input; visual real-time only.
- Training limited to 30-second single-exercise segments; multi-exercise workouts left to future work; possible annotator errors and demographic bias.
Experimental setup
Datasets: QEVD-FIT-300K QEVD-FIT-COACH ImageNet
Benchmarks: QEVD-FIT-COACH
Metrics: METEOR ROUGE-L BERT score LLM-Accuracy temporal F-score
Baselines: InstructBLIP Video-LLaVA Video-ChatGPT Video-LLaMA LLaMA-VID LLaVA-NeXT Socratic-LLaMA-2-7B
Builds on: Video-ChatGPT LLaMA-2 LLaMA-VID Flamingo-style cross-attention
Closest papers by meaning
- ULTRAFEEDBACK: Boosting Language Models with Scaled AI Feedback (ICML 2024 · similarity 0.85)
- LifeEval: A Multimodal Benchmark for Assistive AI in Egocentric Daily Life Tasks (CVPR 2026 · similarity 0.85)
- ExpertAF: Expert Actionable Feedback from Video (CVPR 2025 · similarity 0.85)
- From 3D Pose to Prose: Biomechanics-Grounded Vision-Language Coaching (CVPR 2026 · similarity 0.85)
- OdysSim: Building Foundation Models for Human Behavior Simulation (NeurIPS 2026 · similarity 0.84)
- Can Multi-Modal LLMs Provide Live Step-by-Step Task Guidance? (NeurIPS 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.