Research map / Vision-language and multimodal / Multimodal LLMs

From 3D Pose to Prose: Biomechanics-Grounded Vision-Language Coaching

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Multimodal LLM benchmarks · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes BioCoach, a biomechanics-grounded VLM for streaming fitness coaching that fuses visual features with 3D skeletal kinematics, morphometrics, and cycle/constraint analysis, and releases a re-annotated QEVD-bio-fit-coach benchmark.

Problem

Streaming fitness coaching from video with existing VLMs yields generic, mistimed feedback lacking biomechanical grounding, personalization to body geometry, and explainable rule-level evidence.

Approach

Exposes explicit intermediate representations to the LLM: an exercise-specific DoF selector, individualized morphometric context from SMPL shape via virtual measurements, and motion-quality context from cycle detection and reference alignment, fused with visual tokens via cross-attention while freezing the vision and language backbones.

Key results

• On QEVD-bio-fit-coach, BioCoach reaches METEOR 0.312 vs 0.086 for Stream-VLM (+262.8%) and LLM-Bio-Acc. 3.26 vs 1.72 (+89.5%).
• On original QEVD-fit-coach, BioCoach gets METEOR 0.129, ROUGE-L 0.122, BERTScore 0.864, LLM-Acc. 2.56 vs Stream-VLM 0.127/0.112/0.863/2.45.
• T-F-Score on original QEVD-fit-coach is 0.544 vs Stream-VLM 0.56, near parity in timing.

Surprising findings

• Removing the motion quality context reduces METEOR from 0.312 to 0.133 but leaves T-F-Score unchanged, while the 2 s window keeps text quality but drops T-F-Score by about 24%.
• On the original QEVD-fit-coach annotations, BioCoach has slightly lower T-F-Score than Stream-VLM (0.544 vs 0.56).

Limitations

• Depends on quality of 3D skeleton and shape estimates, which occlusion, loose clothing, and viewpoint can distort.
• Relies on curated reference trajectories that may not cover exercise variants or adaptive forms.
• Angle-only analysis cannot capture kinetics or muscle activation; kinetic reasoning is future work.

Experimental setup

Datasets: QEVD-fit-coach QEVD-bio-fit-coach

Benchmarks: QEVD-bio-fit-coach QEVD-fit-coach

Metrics: METEOR ROUGE-L BERTScore LLM-Accuracy LLM-Bio-Accuracy T-F-Score

Baselines: Stream-VLM Socratic-LLaMA-2-7B Video-ChatGPT LLaMA-VID InstructBLIP Video-LLaVA Video-LLaMA LLaVA-NeXT

Builds on: Stream-VLM QEVD SKEL HSMR Virtual Measurements

Closest papers by meaning

• ExpertAF: Expert Actionable Feedback from Video (CVPR 2025 · similarity 0.86)
• What to Say and When to Say it: Live Fitness Coaching as a Testbed for Situated Interaction (NeurIPS 2024 · similarity 0.85)
• HMVLM:Human Motion-Vision-Language Model via MoE LoRA (NeurIPS 2025 · similarity 0.84)
• Shape My Moves: Text-Driven Shape-Aware Synthesis of Human Motions (CVPR 2025 · similarity 0.83)
• Universal Skeleton Understanding via Differentiable Rendering and MLLMs (ICML 2026 · similarity 0.83)
• CigTime: Corrective Instruction Generation Through Inverse Motion Editing (NeurIPS 2024 · similarity 0.83)