Research map / Image generation / Editing and style transfer
CigTime: Corrective Instruction Generation Through Inverse Motion Editing
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CigTime, which builds source/target/instruction motion triplets by editing HumanML3D motions with MDM and fine-tunes Llama-3-8B over VQ-VAE motion tokens to generate corrective text instructions.
Problem
Generating text instructions that tell a user how to correct their current motion toward a desired target motion, a task lacking annotated triplet data.
Approach
Treat corrective instruction generation as the inverse of text-conditioned motion editing: use a pretrained motion editor (MDM) to create target motions from instructions to build triplets, then fine-tune an LLM on VQ-VAE motion tokens of source and target with an anchor loss keeping original vocabulary embeddings trainable.
Key results
- On HumanML3D, CigTime reaches BLEU 0.24, ROUGE 0.35, METEOR 0.52, CLIPScore 0.82 and MPJPE 0.13 / FID 1.44, beating Llama-3-8B and Mistral-7B baselines.
- On KIT, CigTime achieves MPJPE 0.21 and FID 4.52, outperforming Llama-3-8B-LoRA, Qwen-1.5-7B-LoRA and Mistral-7B-LoRA on reconstruction.
- On out-of-domain Fit3D, text metrics drop sharply (BLEU 0.03) while CLIPScore 0.81 and FID 1.24 remain close to in-domain values.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use CigTime instead of Mistral-7B in-context learning | corrective instruction generation · HumanML3D | BLEU: 0.16 -> 0.24 | Helped |
| Use CigTime instead of Mistral-7B in-context learning | corrective instruction generation · HumanML3D | FID: 5.03 -> 1.44 | Helped |
| Use CigTime instead of Qwen-1.5-7B-LoRA | corrective instruction generation · KIT | MPJPE: 0.33 -> 0.21 | Helped |
| Extend vocabulary with new motion-token embeddings (ablation) | CigTime · HumanML3D | MPJPE: 0.13 -> 0.16 | Hurt |
| Use continuous motion representations (ablation) | CigTime · HumanML3D | FID: 1.44 -> 2.56 | Hurt |
| Use T5 backbone (ablation) | CigTime · HumanML3D | MPJPE: 0.13 -> 0.33 | Hurt |
| Use T5 backbone (ablation) | CigTime · HumanML3D | FID: 1.44 -> 5.03 | Hurt |
| Evaluate on Fit3D instead of HumanML3D | CigTime · Fit3D | BLEU: 0.24 -> 0.03 | Hurt |
| Evaluate on Fit3D instead of HumanML3D | CigTime · Fit3D | MPJPE: 0.13 -> 0.18 | Hurt |
Surprising findings
- Extending the LLM vocabulary with new motion-token embeddings improves text metrics but worsens motion reconstruction accuracy, so the authors prioritize reconstruction over instruction wording.
- Text-overlap metrics (BLEU, ROUGE, METEOR) fall sharply on Fit3D even though reconstruction metrics stay close, suggesting different instructions can yield similar corrected motions.
Limitations
- Curated dataset lacks targeted feedback on sport-specific form and dynamics.
- Limited by the pretrained motion editor to source and target motions of the same sequence length, without context or scenes.
- Could be misused to generate instructions for insulting or inappropriate motions.
Experimental setup
Datasets: HumanML3D KIT Fit3D
Benchmarks: HumanML3D KIT Fit3D
Metrics: BLEU ROUGE METEOR CLIPScore MPJPE FID
Baselines: Llama-3-8B Llama-3-8B-LoRA Mistral-7B Mistral-7B-LoRA MotionGPT MotionGPT-M2T Qwen-1.5-7B-LoRA T5-770M MotionLLM
Builds on: MDM VQ-VAE MotionGPT LoRA HumanML3D
Closest papers by meaning
- AToM: Aligning Text-to-Motion Model at Event-Level with GPT-4Vision Reward (CVPR 2025 · similarity 0.86)
- MG-MotionLLM: A Unified Framework for Motion Comprehension and Generation across Multiple Granularities (CVPR 2025 · similarity 0.86)
- Motion-Agent: A Conversational Framework for Human Motion Generation with LLMs (ICLR 2025 · similarity 0.86)
- ReMoGPT: Part-Level Retrieval-Augmented Motion-Language Models (AAAI 2025 · similarity 0.86)
- OMG: Towards Open-vocabulary Motion Generation via Mixture of Controllers (CVPR 2024 · similarity 0.86)
- DartControl: A Diffusion-Based Autoregressive Motion Model for Real-Time Text-Driven Motion Control (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.