Research map / Vision-language and multimodal / Human and embodied modeling
Semantics-aware Motion Retargeting with Vision-Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SMT, a two-stage motion retargeting method that pre-trains a skeleton-aware graph network unsupervisedly, then fine-tunes it per character pair by aligning BLIP-2 latent embeddings of rendered motion images (guided VQA) plus an interpenetration loss.
Problem
Motion retargeting between animation characters often loses high-level motion semantics and produces interpenetration, while prior methods rely on joint-level representations and labelled semantic data is scarce.
Approach
Use a frozen vision-language model (BLIP-2) as an unsupervised semantic supervisor: differentiably skin and render retargeted motions into images, extract latent embeddings via guiding visual question answering, and enforce a semantics consistency loss between source and target embeddings, after skeleton-level pre-training.
Key results
- On Mixamo, SMT achieves interpenetration 3.50% vs 7.62% for R2ET and ITM 0.680 vs 0.643, with FID 0.436 vs 5.469.
- Ablations show that removing skeleton-aware pre-training or using only one stage degrades MSE and semantics; using image features instead of LLM-encoder features worsens results (SMTfwi FID 78.46).
- User study over 100 comparisons: semantics score 0.92 for SMT vs 0.85 for R2ET.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use SMT instead of R2ET | Motion retargeting on Mixamo · Mixamo test motions | PEN.%: 7.62 -> 3.50 | Helped |
| use SMT instead of R2ET | Motion retargeting on Mixamo · Mixamo test motions | ITM: 0.643 -> 0.680 | Helped |
| use skeleton-aware pre-training (ablation) | SMT · Mixamo test motions | MSE: 7.798 -> 0.284 | Helped |
| use semantics and geometry fine-tuning (ablation) | SMT · Mixamo test motions | PEN.%: 8.37 -> 3.50 | Helped |
| use two-stage training instead of one-stage training (ablation) | SMT · Mixamo test motions | MSE: 0.335 -> 0.284 | Helped |
| add semantics consistency loss alongside interpenetration loss (ablation) | SMT fine-tuning · Mixamo test motions | PEN.%: 1.22 -> 3.50 | Hurt |
| use large-language-model encoder features instead of image features (ablation) | SMT semantic embedding · Mixamo test motions | MSE: 5.418 -> 0.284 | Helped |
| use large-language-model encoder features instead of querying-transformer features (ablation) | SMT semantic embedding · Mixamo test motions | MSE: 0.739 -> 0.284 | Helped |
Surprising findings
- The Copy baseline achieves the lowest local MSE because Mixamo ground truth is not clean and many samples are generated by copying rotations.
- Fine-tuning with only semantics and geometry losses (no pre-training) gives much worse MSE and semantics, attributed to the non-linearity of the semantics loss.
Limitations
- VLM performance on motion semantics is limited since no sufficiently large motion-semantic datasets exist.
- Projecting 3D motion to 2D images loses spatial information (depth).
- Mixamo ground truth is noisy, requiring manual selection of clean subsets.
Experimental setup
Datasets: Mixamo Human3.6M
Benchmarks: Mixamo
Metrics: MSE local MSE interpenetration percentage ITM score FID semantics consistency loss user study preference
Baselines: Copy NKN SAN R2ET
Builds on: BLIP-2 Villegas et al. unsupervised motion retargeting (NKN) Aberman et al. skeleton-aware networks (SAN) R2ET residual retargeting linear blend skinning
Closest papers by meaning
- Skinned Motion Retargeting With Residual Perception of Motion Semantics & Geometry (CVPR 2023 · similarity 0.89)
- HMVLM:Human Motion-Vision-Language Model via MoE LoRA (NeurIPS 2025 · similarity 0.86)
- ReMoGPT: Part-Level Retrieval-Augmented Motion-Language Models (AAAI 2025 · similarity 0.86)
- LaMP: Language-Motion Pretraining for Motion Generation, Retrieval, and Captioning (ICLR 2025 · similarity 0.85)
- SimMotionEdit: Text-Based Human Motion Editing with Motion Similarity Prediction (CVPR 2025 · similarity 0.85)
- MMM: Generative Masked Motion Model (CVPR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.