Research map / Multi-agent systems and games / LLM agents

DMT-RoleBench: A Dynamic Multi-Turn Dialogue Based Benchmark for Role-Playing Evaluation of Large Language Model and Agent

AAAI 2025 · Multi-agent systems and games › LLM agents · approach: Agent data synthesis and behavior studies · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces DMT-RoleBench, a Chinese role-playing benchmark that dynamically generates multi-turn dialogues guided by evaluation intents, scores them with a fine-tuned judge (DMT-RM) and a new DMT-Score, and evaluates seven LLMs/APIs.

Problem

Existing role-playing benchmarks rely on multiple-choice knowledge questions or pre-defined dialogue completion, which do not reflect real multi-turn user interactions with role-playing agents.

Approach

Dynamically generate multi-turn dialogues with a generator LLM that acts as user, constrained by role profile, topic and evaluation intent; annotate each round as good/bad with a fine-tuned judge (DMT-RM, Qwen2-7B-Instruct backbone) and aggregate with DMT-Score, which applies strict or any-round scoring depending on the metric.

Key results

• Doubao-Pro-32K achieves the best results on most of the 14 metrics, e.g., 88.65 on Instruction Following and 99.49 on Role Embodying.
• DMT-RM reports over 95% consistency with crowd-worker annotations and is reported to outperform GPT-4-Turbo as an annotator.
• All models degrade as dialogue rounds increase, with GPT-3.5 showing the steepest decline; all models struggle on Game Completion Degree (GPT series best at 64.68 for GPT4 and 63.04 for GPT-4o).

Measured effects reported in the paper

Surprising findings

• GPT-4o shows inferior diversity relative to other models due to reliance on similar sentence structures.
• GPT-series models perform better with the Minimalist system prompt than the Overview one, attributed partly to perturbation from role type and intent distribution.

Limitations

• Seed data is Chinese-only, limiting generality across languages.
• Judge DMT-RM consistency is reported only in aggregate and details are in an external repository.
• Evaluated models accessed only via APIs, and the 996-sample seed set is relatively small.

Experimental setup

Datasets: DMT-RoleBench seed dataset (996 samples) crowd-worker annotations

Benchmarks: DMT-RoleBench

Metrics: DMT-Score consistency with human annotation good recall badcase recall

Baselines: GPT-4-Turbo GPT-4o GPT-3.5-Turbo Qwen2-72B-Instruct Doubao-Pro-32K ErnieBot-4.0 GLM-4-0520

Builds on: CharacterEval RoleEval Role-LLM

Closest papers by meaning

• RoleAgent: Building, Interacting, and Benchmarking High-quality Role-Playing Agents from Scripts (NeurIPS 2024 · similarity 0.87)
• MMRole: A Comprehensive Framework for Developing and Evaluating Multimodal Role-Playing Agents (ICLR 2025 · similarity 0.87)
• AgentBench: Evaluating LLMs as Agents (ICLR 2024 · similarity 0.86)
• CharacterBench: Benchmarking Character Customization of Large Language Models (AAAI 2025 · similarity 0.85)
• LMRL Gym: Benchmarks for Multi-Turn Reinforcement Learning with Language Models (ICML 2025 · similarity 0.85)
• Codifying Character Logic in Role-Playing (NeurIPS 2025 · similarity 0.85)