Research map / Multi-agent systems and games / Human-AI and goal-conditioned
Advancing DRL Agents in Commercial Fighting Games: Training, Integration, and Agent-Human Alignment
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Presents Shūkai, a unified DRL agent system for the commercial fighting game Naruto Mobile, using heterogeneous league training (HELT) and quantified states plus reward-based style control to generalize across characters and align behavior with human players.
Problem
Training DRL agents for a fighting game with over 400 characters is computationally costly, subset training overfits and fails to generalize to unseen characters, and agents tuned for maximum competence can frustrate average players.
Approach
Combines three heterogeneous network structures (FIS, QS, FQS) with different league roles in an AlphaStar-style league (main agent, main exploiter, league exploiter), so quantified-state agents generalize while ID-based agents add competence; adds style-specific rewards and action masks/delays to produce human-like agents at several difficulty levels.
Key results
- The FIS agent's competence dropped by 36% on unfamiliar characters while QS and FQS remained stable.
- After 70 hours of training, HELT achieved a 22% competence improvement over QS with homogeneous league training.
- Deployed to Naruto Mobile, Shūkai reported a maximum 5% and average 4% retention growth in an A/B test, with next-day retention above 50%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use HELT instead of homogeneous league training | QS agent · Naruto Mobile, 70 hours | competence: 22% | Helped |
| Quantify opponent state inputs rather than use full ID information | DRL agent · Naruto Mobile, unfamiliar characters | competence: FIS competence dropped by 36%; QS and FQS remained stable | Helped |
| Use ID information to model opponents | DRL agent · Naruto Mobile, familiar characters | competence | Helped |
| Increase state quantization from QS to FQS | DRL agent · Naruto Mobile, familiar characters | competence | Hurt |
| Train with Shūkai beginner as a player training partner | Naruto Mobile players · Naruto Mobile, after 30 days | Elo score: 9% decrease | Helped |
| Use aggressive rather than cautious rewards | Shūkai behavior · Naruto Mobile, Counter metric | behavior reward | Hurt |
| Use balanced rewards rather than aggressive or cautious rewards | Shūkai behavior · Naruto Mobile, Counter metric | behavior reward | Mixed |
Surprising findings
- The ID-based FIS agent is strongest on familiar characters but loses 36% competence on unfamiliar ones, while quantified-state agents generalize better.
- Shūkai agents were found to be more cautious and more precise in skill release than human players.
Limitations
- Evaluation of generalization is limited to one game (Naruto Mobile) and a 50-character training subset.
- Agents remain more cautious than humans and lack high reward-score behaviors, indicating a remaining gap.
- Retention results are from a proprietary commercial A/B test that cannot be independently verified.
Experimental setup
Datasets: Naruto Mobile gameplay data (40 million matches for analysis; 120,000 game instances for behavior comparison)
Benchmarks: Naruto Mobile matches against oracle agent Naruto Mobile live deployment (player Elo and retention A/B test)
Metrics: Elo score win rate sample efficiency player retention rate CDF of behavior metrics (Substitution, Special, Blitz, Count
Baselines: homogeneous league training (HOLT) FIS agent QS agent FQS agent
Builds on: AlphaStar PPO prioritized fictitious self-play generalized advantage estimation
Closest papers by meaning
- FightLadder: A Benchmark for Competitive Multi-Agent Reinforcement Learning (ICML 2024 · similarity 0.87)
- A Generalist Hanabi Agent (ICLR 2025 · similarity 0.85)
- Dr. MAS: Stable Reinforcement Learning for Multi-Agent LLM Systems (NeurIPS 2026 · similarity 0.85)
- JaxMARL: Multi-Agent RL Environments and Algorithms in JAX (NeurIPS 2024 · similarity 0.84)
- Cheaper and Faster: Distributed Deep Reinforcement Learning with Serverless Computing (AAAI 2024 · similarity 0.84)
- DistRL: An Asynchronous Distributed Reinforcement Learning Framework for On-Device Control Agent (ICLR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.