Research map / Vision-language and multimodal / Human and embodied modeling
AToM: Aligning Text-to-Motion Model at Event-Level with GPT-4Vision Reward
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds MotionPrefer, a GPT-4V-annotated event-level text-motion preference dataset, and fine-tunes MotionGPT with LoRA and IPO to improve integrity, temporal-order, and frequency alignment in text-to-motion generation.
Problem
Text-to-motion models struggle to align generated motion with event-level textual descriptions involving multiple events, temporal order, and frequency, and existing preference data is small, human-labeled, or coarse-grained.
Approach
Use GPT-4V as a fine-grained AI reward on rendered 8-frame motion sequences, with task-specific scoring rules for integrity, temporal order, and frequency, to create preference pairs and fine-tune a motion generator via LoRA with IPO loss.
Key results
- On the temporal subset, AToM improves FID from 0.655 to 0.613 and top-1 from 0.193 to 0.199 over MotionGPT.
- In a 50-participant user study, AToM win rates over MotionGPT were 74.4% (temporal), 70.0% (frequency), and 84.4% (integrity).
- On the general task, AToM achieves MM Dist 3.943 and FID 0.177, versus InstructMotion's 3.947 and 0.260.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune MotionGPT with GPT-4V preference feedback | MotionGPT · Temporal | MM Dist: 5.652±.048 -> 5.576±.027 | Helped |
| Fine-tune MotionGPT with GPT-4V preference feedback | MotionGPT · Temporal | Top-1: 0.193±.004 -> 0.199±.005 | Helped |
| Fine-tune MotionGPT with GPT-4V preference feedback | MotionGPT · Frequency | MM Dist: 5.328±.063 -> 5.259±.045 | Helped |
| Fine-tune MotionGPT with GPT-4V preference feedback | MotionGPT · Frequency | Top-1: 0.182±.011 -> 0.183±.009 | No clear effect |
| Fine-tune MotionGPT with GPT-4V preference feedback | MotionGPT · Integrity | MM Dist: 5.897±.036 -> 5.871±.034 | Helped |
| Fine-tune MotionGPT with GPT-4V preference feedback | MotionGPT · Integrity | Top-1: 0.160±.007 -> 0.160±.007 | No clear effect |
| Fine-tune with a mixed, approximately 3.5K-pair preference subset | MotionGPT · General | Top-1: 0.407±.002 -> 0.410±.008; 0.402±.003 -> 0.410±.008 | Helped |
| Use a composite image of sampled frames instead of frame-by-frame images (ablation) | AToM · Test set | MM Dist: 5.576±.027 -> 5.657±.048 | Hurt |
| Represent motion frames as rigged cylinders in one image (ablation) | AToM · Test set | MM Dist: 5.576±.027 -> 5.635±.036 | Hurt |
| Remove score filtering when constructing preference pairs (ablation) | AToM · Test set | MM Dist: 5.576±.027 -> 5.640±.030 | Hurt |
| Add LoRA to fine-tuning (ablation) | MotionGPT · Test set | MM Dist: 6.425±.066 -> 5.576±.027 | Helped |
| Use IPO rather than DPO, KTO, or PPO (ablation) | MotionGPT fine-tuning · Fine-tuning evaluation | Overall performance | Helped |
Surprising findings
- Full-image and trajectory-image motion injection performed worse than frame-by-frame sequences for GPT-4V scoring.
- Among RL strategies, IPO outperformed DPO, KTO, and PPO after fine-tuning.
Limitations
- Gains on some metrics are mixed; diversity and MModality decline slightly after fine-tuning.
- Evaluation relies on a single base generator (MotionGPT) and on GPT-4V scores that are not validated beyond a user study.
Experimental setup
Datasets: HumanML3D MotionPrefer
Benchmarks: HumanML3D test set (filtered subsets for integrity, temporal, frequency)
Metrics: MM Dist R-Precision Top-1/2/3 FID Diversity MModality win rate
Baselines: MotionGPT InstructMotion
Builds on: MotionGPT IPO LoRA InstructMotion
Closest papers by meaning
- ReAlign: Text-to-Motion Generation via Step-Aware Reward-Guided Alignment (AAAI 2026 · similarity 0.90)
- SnapMoGen: Human Motion Generation from Expressive Texts (NeurIPS 2025 · similarity 0.87)
- DartControl: A Diffusion-Based Autoregressive Motion Model for Real-Time Text-Driven Motion Control (ICLR 2025 · similarity 0.87)
- OMG: Towards Open-vocabulary Motion Generation via Mixture of Controllers (CVPR 2024 · similarity 0.87)
- ReMoGPT: Part-Level Retrieval-Augmented Motion-Language Models (AAAI 2025 · similarity 0.86)
- CigTime: Corrective Instruction Generation Through Inverse Motion Editing (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.