Research map / Reinforcement learning / Imitation learning
Generative Adversarial Post-Training Mitigates Reward Hacking in Live Human-AI Music Interaction
Yusong Wu, Stephen Brade, Teng Ma, Tia-Jane Fowler, Enning Yang, Berker Banar, Aaron Courville, Natasha Jaques, Cheng-Zhi Anna Huang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add adversarial reward to RL post-training (PPO) (ablation) | GAPT · fixed-melody test set | diversity (Vendi): 20.814 -> 26.645 | Improved |
| add adversarial reward to RL post-training (PPO) (ablation) | GAPT · fixed-melody test set | harmony: 0.476 -> 0.497 | Improved |
| add adversarial reward to RL post-training (PPO) (ablation) | GAPT · out-of-distribution Wikifonia | diversity (Vendi): 8.034 -> 11.295 | Improved |
| add adversarial training to GRPO (ablation) | GRPO · fixed-melody test set | diversity (Vendi): 16.872 -> 26.603 | Improved |
| add adversarial training to GRPO (ablation) | GRPO · fixed-melody test set | harmony: 0.459 -> 0.478 | Improved |
| discriminator conditioned on melody and chord instead of chord only (ablation) | GAPT · fixed-melody test set | diversity (Vendi): 26.645 -> 23.545 | Hurt |
| upweight rule penalties (beta=2) (ablation) | GAPT · fixed-melody test set | harmony and diversity: 0.497 -> 0.475; 26.645 -> 25.667 | Hurt |
| upweight coherence reward (alpha=2) (ablation) | GAPT · fixed-melody test set | harmony and diversity: 0.497 -> 0.494; 26.645 -> 26.742 | No clear effect |
| upweight adversarial reward (gamma=2) (ablation) | GAPT · fixed-melody test set | harmony: 0.497 -> 0.456 | Hurt |
| remove rule-based penalties entirely (ablation) | GAPT · fixed-melody test set | harmony and diversity | Hurt |
| replace Online MLE by GAPT as accompaniment policy | live jamming with musicians · user study (12 musicians) | adaptation speed and control/agency ratings: p<0.05 vs ReaLchords | Improved |
| use GAPT instead of Online MLE with a learned melody agent | GAPT · learned melody agent | harmony and diversity: diversity 18.071 -> 12.914 | Hurt |
Closest papers by meaning
- Adaptive Accompaniment with ReaLchords (ICML 2024 · similarity 0.90)
- Live Music Diffusion Models: Efficient Fine-Tuning and Post-Training of Interactive Diffusion Music Generators (NeurIPS 2026 · similarity 0.86)
- CMI-RewardBench: Evaluating Music Reward Models with Compositional Multimodal Instruction (ICML 2026 · similarity 0.85)
- MusicRL: Aligning Music Generation to Human Preferences (ICML 2024 · similarity 0.85)
- JEN-1 Composer: A Unified Framework for High-Fidelity Multi-Track Music Generation (AAAI 2025 · similarity 0.84)
- Compose with Me: Collaborative Music Inpainter for Symbolic Music Infilling (AAAI 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
