Research map / Vision-language and multimodal / Human and embodied modeling
ReMoGPT: Part-Level Retrieval-Augmented Motion-Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ReMoGPT, a unified motion-language model that augments T5-based MotionGPT-style instruction tuning with retrieved motion-caption pairs from a body-part-level text-motion retrieval model (PL-TMR).
Problem
Motion-language models such as MotionGPT struggle with rare or unconventional text conditions and diverse motions, while retrieval-augmented motion generation (ReMoDiffuse) relies only on text-to-text similarity and cannot caption motions.
Approach
Retrieve references with a body-part-level text-motion retriever (PL-TMR) that encodes six body parts separately, then include retrieved motion tokens and captions in the prompt of a T5-based motion-language model for both text-to-motion generation and motion captioning.
Key results
- On HumanML3D text-to-motion generation, ReMoGPT reaches FID 0.205 and R-Top1 0.501, versus 0.361 FID and 0.431 R-Top1 for re-trained MotionGPT.
- On HumanML3D motion-to-text captioning, ReMoGPT achieves BLEU@4 13.4 and CIDEr 31.5, above MotionGPT's 12.4 and 29.3.
- PL-TMR retrieval on HumanML3D reaches R@1 11.00 for text-to-motion, above MotionPatches (10.80) and TMR (8.92), with 118M parameters.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use body-part-level PL-TMR instead of MotionPatches | HumanML3D text-to-motion retrieval · HumanML3D | R@1: 10.80 -> 11.00 | Helped |
| add retrieval augmentation to MotionGPT | HumanML3D text-to-motion generation · HumanML3D | RPrecision Top1: 0.431 -> 0.501 | Helped |
| remove retrieval augmentation (ablation) | ReMoGPT · HumanML3D text-to-motion generation | R-Top1: 0.431 -> 0.501 | Hurt |
| add text-to-text retrieval to the no-retrieval model (ablation) | ReMoGPT · HumanML3D text-to-motion generation | R-Top1: 0.431 -> 0.472 | Helped |
| add text-to-motion retrieval to the no-retrieval model (ablation) | ReMoGPT · HumanML3D text-to-motion generation | R-Top1: 0.431 -> 0.479 | Helped |
| use both text-to-text and text-to-motion retrieval instead of either single modality (ablation) | ReMoGPT · HumanML3D text-to-motion generation | R-Top1: 0.479 -> 0.501 | Helped |
| increase retrieved samples from 1+1 to 2+2 (ablation) | ReMoGPT · HumanML3D text-to-motion generation | R-Top1: 0.501 -> 0.498 | No clear effect |
| replace PL-TMR with TMR for retrieval (ablation) | ReMoGPT · HumanML3D text-to-motion generation | R-Top1: 0.501 -> 0.493 | Hurt |
| replace PL-TMR with MotionPatches for retrieval (ablation) | ReMoGPT · HumanML3D text-to-motion generation | R-Top1: 0.501 -> 0.497 | Hurt |
| replace the HumanML3D retrieval database with Motion-X (ablation) | ReMoGPT · HumanML3D text-to-motion generation | FID: 0.205 -> 0.189 | Helped |
| replace the Motion-X retrieval database with HumanML3D (ablation) | ReMoGPT · Motion-X text-to-motion generation | FID: 0.352 -> 0.608 | Hurt |
| add retrieval augmentation for rare-motion generation | ReMoGPT · HumanML3D rare text-to-motion generation | MMDist Top 5%: 4.421 -> 3.563 | Helped |
Surprising findings
- Using four retrieved samples instead of two gave only limited gains, which the authors attribute to T5 struggling with very long motion-token inputs.
Limitations
- Struggles to generate completely new motions far from the training distribution (e.g., yoga) because the motion tokenizer generalizes poorly.
- Using four retrieved samples gives limited gains due to T5 handling long motion-token sequences.
- Retrieval database is the training set; cross-dataset transfer degrades performance on Motion-X when HumanML3D is the database.
Experimental setup
Datasets: HumanML3D Motion-X KIT
Benchmarks: HumanML3D Motion-X
Metrics: R-Precision FID MM Dist Diversity MultiModality Recall@K MedR BLEU@4 ROUGE CIDEr
Baselines: TM2T T2M MotionDiffuse MDM MLD T2M-GPT ReMoDiffuse MotionGPT TMR MotionPatches
Builds on: MotionGPT ReMoDiffuse T5 VQ-VAE TMR
Closest papers by meaning
- VimoRAG: Video-based Retrieval-augmented 3D Motion Generation for Motion Language Models (NeurIPS 2025 · similarity 0.91)
- SnapMoGen: Human Motion Generation from Expressive Texts (NeurIPS 2025 · similarity 0.89)
- AMD: Autoregressive Motion Diffusion (AAAI 2024 · similarity 0.89)
- M$^3$GPT: An Advanced Multimodal, Multitask Framework for Motion Comprehension and Generation (NeurIPS 2024 · similarity 0.89)
- The Language of Motion: Unifying Verbal and Non-verbal Language of 3D Human Motion (CVPR 2025 · similarity 0.89)
- LaMP: Language-Motion Pretraining for Motion Generation, Retrieval, and Captioning (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.