Research map / Vision-language and multimodal / Human and embodied modeling
Exploring Vision Transformers for 3D Human Motion-Language Models with Motion Patches
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes "motion patches" (joints grouped into five body parts, interpolated to N points, stacked over N frames) so that an ImageNet-pretrained ViT-B/16 can serve as the motion encoder in a CLIP-style motion-text retrieval model.
Problem
Motion-language models are limited by scarce 3D motion-text data and by skeleton-specific input formats that prevent unified training across datasets.
Approach
Convert skeleton joint sequences into fixed-size image-like patches (body-part-wise, linearly interpolated, z-scored, joint xyz treated as RGB) so that ImageNet-pretrained ViT weights can be transferred to motion encoding, yielding a skeleton-agnostic representation.
Key results
- On HumanML3D (All protocol), text-to-motion R@1 improves from 8.92 (TMR) to 10.80 and R@10 from 33.37 to 38.02.
- On KIT-ML, transfer from HumanML3D lifts text-to-motion R@1 to 15.28 versus 10.05 for TMR trained on KIT-ML.
- Zero-shot BABEL-60 classification reaches Top-1 41.33%, comparable to supervised 2s-AGCN (41.14%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use pre-trained ViT weights rather than training from scratch (ablation) | Motion-patch motion-language model · HumanML3D, text-to-motion | R@1: 8.46 -> 10.80 | Helped |
| Use motion patches instead of raw motion sequences (ablation) | Pre-trained ViT motion-language model · HumanML3D, text-to-motion | R@1: 8.36 -> 10.80 | Helped |
| Use pre-trained ViT weights rather than training from scratch (ablation) | Motion-patch motion-language model · KIT-ML, text-to-motion | R@1: 10.41 -> 14.02 | Helped |
| Use motion patches instead of raw motion sequences (ablation) | Pre-trained ViT motion-language model · KIT-ML, text-to-motion | R@1: 9.54 -> 14.02 | Helped |
| Fine-tune the HumanML3D model on KIT-ML | Ours · KIT-ML, cross-skeleton text-to-motion retrieval | R@1: 7.35 -> 15.28 | Helped |
| Use the motion-language model for zero-shot classification | HumanML3D-trained Ours · BABEL-60 | Top-1 accuracy: 30.13 -> 41.33 | Helped |
| Use motion patches and ViT for interaction recognition | Ours · InterHuman, text-to-motion | R@1: 5.38 -> 9.51 | Helped |
Surprising findings
- Ablation shows ViT pretraining gives only modest gains over scratch training on HumanML3D (e.g., R@1 10.80 vs 8.46 with patches) while the motion-patch input matters more than pretraining in some metrics.
- Text encoder choice: DistilBERT outperformed CLIP's text encoder in the authors' experiments.
Limitations
- Evaluated mainly on retrieval and recognition; text-to-motion generation not explored.
- Generalization may be limited by small motion-text data relative to image-text data.
- Gains over scratch training on HumanML3D are modest in several metrics.
Experimental setup
Datasets: HumanML3D KIT Motion-Language BABEL InterHuman ImageNet-21k
Benchmarks: HumanML3D KIT-ML BABEL-60 InterHuman
Metrics: Recall@K MedR Top-1 accuracy Top-5 accuracy
Baselines: TEMOS T2M TMR MotionCLIP 2s-AGCN Ours (scratch)
Builds on: CLIP ViT TMR TEMOS DistilBERT
Closest papers by meaning
- Tri-Modal Motion Retrieval by Learning a Joint Embedding Space (CVPR 2024 · similarity 0.87)
- Part-aware Unified Representation of Language and Skeleton for Zero-shot Action Recognition (CVPR 2024 · similarity 0.87)
- SAMoR: Motion Modelling for Articulated Objects of Any Skeleton and Topology (NeurIPS 2026 · similarity 0.87)
- SimMotionEdit: Text-Based Human Motion Editing with Motion Similarity Prediction (CVPR 2025 · similarity 0.86)
- The Language of Motion: Unifying Verbal and Non-verbal Language of 3D Human Motion (CVPR 2025 · similarity 0.86)
- HMVLM:Human Motion-Vision-Language Model via MoE LoRA (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.