Research map / Video / Video generation
LaMP: Language-Motion Pretraining for Motion Generation, Retrieval, and Captioning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LaMP, a language-motion pretraining model (BERT-initialized dual transformers with four alignment objectives) whose text features replace CLIP as conditions for a decoder-only masked motion transformer, and whose motion features support retrieval and LLM-based captioning.
Problem
CLIP text embeddings, pretrained on static image-text pairs, poorly align language with human motion dynamics, limiting text-to-motion generation, motion-text retrieval, and motion captioning.
Approach
Replace the CLIP language-vision latent space with a language-motion space learned via joint contrastive, matching, motion-grounded text generation, and text-grounded motion generation objectives, then use its text features as generation conditions and its motion features for retrieval and captioning; generation uses a decoder-only (causal-mask) masked transformer, argued to avoid low-rank rank co
Key results
- LaMP-T2M reaches FID 0.032 on HumanML3D (vs. 0.045 for MoMask) and R-Precision Top-1 0.557 vs. 0.521 for MoMask.
- On KIT-ML, LaMP-T2M reports FID 0.141 and R-Precision Top-1 0.479 vs. 0.433 for MoMask.
- LaMP-M2T achieves R-Precision Top-1 0.547 and BLEU@4 13.04 on HumanML3D captioning, slightly above MotionGPT on most metrics but lower on CIDEr (28.9 vs 29.2) and BLEU@1.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace CLIP text encoder with LaMP (ablation) | LaMP-T2M · HumanML3D | FID: 0.086^{\pm.002} -> 0.040^{\pm.002} | Helped |
| add motion-text matching (ablation) | LaMP pretraining · HumanML3D | R-Precision Top 1: 0.519^{\pm.002} -> 0.547^{\pm.002} | Helped |
| add motion-grounded text generation (ablation) | LaMP pretraining with motion-text matching · HumanML3D | R-Precision Top 1: 0.547^{\pm.002} -> 0.549^{\pm.002} | Helped |
| add text-grounded motion generation (ablation) | LaMP pretraining with the other proxy tasks · HumanML3D | FID: 0.062^{\pm.002} -> 0.032^{\pm.002} | Helped |
| add query-token interaction (ablation) | LaMP-T2M · HumanML3D | FID: 0.040^{\pm.002} -> 0.042^{\pm.002} | Mixed |
| use decoder-only mask transformer (ablation) | LaMP-T2M · HumanML3D | FID: 0.042^{\pm.002} -> 0.032^{\pm.002} | Helped |
| use LaMP-T2M instead of MoMask | text-to-motion generation · HumanML3D | FID: 0.045^{\pm.002} -> 0.032^{\pm.002} | Helped |
| use LaMP-T2M instead of MoMask | text-to-motion generation · KIT-ML | R-Precision Top 1: 0.433^{\pm.007} -> 0.479^{\pm.006} | Helped |
| use LaMP-Feat instead of TMR | text-motion retrieval · KIT-ML | R@1: 49.25 -> 52.5^{\pm0.7} | Helped |
| use LaMP-M2T instead of Motiongpt | motion captioning · HumanML3D | Bleu@4: 12.47 -> 13.04 | Helped |
Surprising findings
- The LaMP-M2T captioning model is slightly worse than MotionGPT on CIDEr (28.9 vs. 29.2) and BLEU@1 (47.8 vs. 48.2), despite the claim of competitive performance.
- Results with the LaMP text encoder trained on HumanML3D and applied to KIT-ML show only a slight decline relative to in-domain training, which the authors report as evidence of generalization.
Limitations
- Limited dataset size restricts generalization, e.g., failures on cartoon or non-human motions such as flying.
- Failure cases stem from noisy textual annotations in the dataset and from unseen motion-text combinations.
Experimental setup
Datasets: HumanML3D KIT-ML
Benchmarks: HumanML3D KIT-ML
Metrics: R-Precision FID MultiModal Distance Diversity LaMP-BertScore Recall@K BLEU@1 BLEU@4 ROUGE CIDEr
Baselines: T2M-GPT MoMask MLD MotionDiffuse TMR T2M MotionGPT TM2T
Builds on: BLIP-2 MoMask T2M-GPT CLIP VQ-VAE
Closest papers by meaning
- ReMoGPT: Part-Level Retrieval-Augmented Motion-Language Models (AAAI 2025 · similarity 0.89)
- VimoRAG: Video-based Retrieval-augmented 3D Motion Generation for Motion Language Models (NeurIPS 2025 · similarity 0.88)
- SnapMoGen: Human Motion Generation from Expressive Texts (NeurIPS 2025 · similarity 0.87)
- MMM: Generative Masked Motion Model (CVPR 2024 · similarity 0.87)
- OMG: Towards Open-vocabulary Motion Generation via Mixture of Controllers (CVPR 2024 · similarity 0.86)
- The Language of Motion: Unifying Verbal and Non-verbal Language of 3D Human Motion (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.