Research map / Reinforcement learning / Visual and representation RL
Explore 3D Dance Generation via Reward Model from Automatically-Ranked Demonstrations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes E3D2, which trains a reward model on automatically ranked dance demonstrations (noise-injected behavior-cloning policies, D-REX style) and fine-tunes a Bailando-style music-to-dance policy with PPO to improve exploration, diversity, and human alignment on AIST++.
Problem
Music-conditioned 3D dance generation models trained by supervised behavior cloning produce monotonous, simplistic dances that generalize poorly to unseen music and misalign with human preferences.
Approach
Learn a reward model from automatically ranked demonstrations generated by injecting increasing epsilon-greedy noise into a behavior-cloning policy (D-REX-style), then use it as a sparse sequence-level reward to fine-tune the dance policy with PPO so it explores beyond the human data distribution.
Key results
- E3D2 reaches FID_k 26.25 and FID_g 8.94 on AIST++ test, versus Bailando's 28.62 and 9.95, an improvement of 8.28% and 10.15%.
- E3D2 improves DIV_k to 7.96 from Bailando's 6.27 (about 21% gain) and DIV_g to 6.49 from 6.22.
- Reward model reaches roughly 97% classification accuracy on both training and test sets, while BC policy complete-pose accuracy on unseen music is only 2.32%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add reinforcement-learning exploration guided by the learned reward model | Bailando · AIST++ test set | FID_k: 28.62 -> 26.25; improves 8.28% | Helped |
| Add reinforcement-learning exploration guided by the learned reward model | Bailando · AIST++ test set | FID_g: 9.95 -> 8.94; improves 10.15% | Helped |
| Add reinforcement-learning exploration guided by the learned reward model | Bailando · AIST++ test set | DIV_k: 6.27 -> 7.96; improves 21.23% | Helped |
| Add reinforcement-learning exploration guided by the learned reward model | Bailando · AIST++ test set | DIV_g: 6.22 -> 6.49; improves 4.16% | Helped |
| Add reinforcement-learning exploration guided by the learned reward model | Bailando · AIST++ test set | BAS: 0.2220 -> 0.2232 | Helped |
| Inject progressively higher ε-greedy noise into the BC policy (ablation) | Bailando · AIST++ training music | FID_k: 13.94 -> 40.45 -> 48.59 -> 53.79 -> 57.18 | Hurt |
| Inject progressively higher ε-greedy noise into the BC policy (ablation) | Bailando · AIST++ training music | Average total reward: 206.31 -> 127.68 -> 52.09 -> -20.24 -> -91.53 | Hurt |
| Use a hand-designed beat-alignment reward during RL fine-tuning (ablation) | Bailando · AIST++ | FID_k: 28.62 -> 45.39 (1M) -> 56.30 (5M) | Hurt |
Surprising findings
- The hand-designed beat-alignment reward degrades FID and diversity as RL steps increase, performing worse than the BC policy on most metrics.
- BC policy performance drops sharply on unseen music (FID_k 8.48 seen vs 28.62 unseen), indicating weak generalization of supervised dance generation.
Limitations
- Evaluated only on AIST++, the single public aligned dance dataset.
- Reward model trained on noise-injected BC rollouts, so ranking reflects noise level rather than human judgments; no human evaluation reported.
- Theoretical guarantee depends on assumptions (bounded reward error, ||w||_1 <= 1) that may not hold in practice.
Experimental setup
Datasets: AIST++
Benchmarks: AIST++
Metrics: FID_k FID_g DIV_k DIV_g BAS
Baselines: FACT Bailando hand-designed beat-alignment reward
Builds on: Bailando D-REX T-REX Decision Transformer PPO
Closest papers by meaning
- Duolando: Follower GPT with Off-Policy Reinforcement Learning for Dance Accompaniment (ICLR 2024 · similarity 0.85)
- MEGADance: Mixture-of-Experts Architecture for Genre-Aware 3D Dance Generation (NeurIPS 2025 · similarity 0.85)
- DanceAnyWay: Synthesizing Beat-Guided 3D Dances with Randomized Temporal Contrastive Learning (AAAI 2024 · similarity 0.85)
- Bidirectional Autoregessive Diffusion Model for Dance Generation (CVPR 2024 · similarity 0.85)
- Controllable 3D Dance Generation Using Diffusion-Based Transformer U-Net (AAAI 2025 · similarity 0.84)
- POPDG: Popular 3D Dance Generation with PopDanceSet (CVPR 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.