Research map / Video / Video-language understanding
Tri-Modal Motion Retrieval by Learning a Joint Embedding Space
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LAVIMO, a tri-modal (text-video-motion) contrastive joint embedding framework that uses rendered human videos as an intermediary modality plus a motion-query attention fusion module, evaluated on HumanML3D and KIT-ML for text-motion and video-motion retrieval.
Problem
Text-to-motion retrieval is limited by scarce motion datasets and the large spatial gap between text and motion modalities.
Approach
Uses human-centric video as an intermediary modality that is closer to motion than text, aligning text, video and motion in one joint space with KL-based contrastive losses, negative filtering from TMR, and a reconstruction branch where motion embeddings query text and video features via attention.
Key results
- On HumanML3D (all protocol), the 3-modal LAVIMO reaches text-to-motion R@1 6.37 vs TMR 5.68, and on the dissimilar subset R@1 50.00 vs TMR 34.00.
- On KIT-ML (all protocol), 3-modal text-to-motion R@1 is 10.16 vs TMR 7.23.
- Video-to-motion retrieval on HumanML3D (all) R@1 is 21.25 vs MotionSet 8.26; a user study reports 68.5% subjective match on 100 real-life videos.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add the video modality to the text-motion model (ablation) | LAVIMO · HumanML3D, All | text-to-motion R@1: 5.93 -> 6.37 | Helped |
| Add the video modality to the text-motion model (ablation) | LAVIMO · KIT-ML, All | text-to-motion R@1: 8.59 -> 10.16 | Helped |
| Add the video modality to the text-motion model (ablation) | LAVIMO · HumanML3D, All | motion-to-text R@1: 8.08 -> 9.72 | Helped |
| Add the video modality to the text-motion model (ablation) | LAVIMO · KIT-ML, All | motion-to-text R@1: 11.72 -> 15.43 | Helped |
| Use LAVIMO instead of TMR | text-to-motion retrieval · HumanML3D, All | text-to-motion R@1: 5.68 -> 6.37 | Helped |
| Use LAVIMO instead of TMR | text-to-motion retrieval · KIT-ML, All | text-to-motion R@1: 7.23 -> 10.16 | Helped |
| Use LAVIMO instead of MotionSet | video-to-motion retrieval · HumanML3D, All | video-to-motion R@1: 8.26 -> 21.25 | Helped |
| Use LAVIMO instead of MotionSet | video-to-motion retrieval · KIT-ML, All | video-to-motion R@1: 9.15 -> 18.75 | Helped |
Surprising findings
- The 2-modal variant underperforms TMR on some HumanML3D protocols (e.g., motion-to-text R@1 8.08 vs 9.95 in the all protocol).
Limitations
- Video modality is rendered from animated avatars rather than authentic human-centric footage.
- Retrieval precision lags video and image retrieval tasks, attributed to limited motion datasets.
- Real-world generalization is evaluated only on a small 100-video user study.
Experimental setup
Datasets: HumanML3D KIT-ML AMASS HumanAct12
Benchmarks: HumanML3D KIT-ML
Metrics: R@1 R@2 R@3 R@5 R@10 MedR
Baselines: TEMOS Guo et al. MotionCLIP TMR MotionSet
Builds on: TMR MotionCLIP CLIP DistilBERT
Closest papers by meaning
- VimoRAG: Video-based Retrieval-augmented 3D Motion Generation for Motion Language Models (NeurIPS 2025 · similarity 0.89)
- Exploring Vision Transformers for 3D Human Motion-Language Models with Motion Patches (CVPR 2024 · similarity 0.87)
- ReMoGPT: Part-Level Retrieval-Augmented Motion-Language Models (AAAI 2025 · similarity 0.87)
- SimMotionEdit: Text-Based Human Motion Editing with Motion Similarity Prediction (CVPR 2025 · similarity 0.87)
- Enhanced Motion-Text Alignment for Image-to-Video Transfer Learning (CVPR 2024 · similarity 0.86)
- Semantics-aware Test-time Adaptation for 3D Human Pose Estimation (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.