Research map / Multi-agent systems and games / LLM agents
RoleAgent: Building, Interacting, and Benchmarking High-quality Role-Playing Agents from Scripts
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RoleAgent, a framework that builds role-playing agents directly from raw TV/film scripts using hierarchical memory and a four-step interaction pipeline, and releases RoleAgentBench (128 roles across 25 scripts) for evaluating them.
Problem
Generative agents need human-annotated profiles for initialization, which limits scaling to many characters; there is no benchmark for evaluating script-derived role-playing agents.
Approach
Automatically derive agent observations from raw scripts, compress them into a three-tier hierarchy (events, key points, insights), and at interaction time deconstruct queries, retrieve memories with a four-factor score, replay raw observations, and summarize subconscious memories and intentions before response generation.
Key results
- On the English subset, RoleAgent with GPT-4 reaches 94.7% self-knowledge accuracy and 88.9% reaction accuracy, with Qwen-Max close behind (94.1%, 92.9%).
- On the English subset, RoleAgent on GPT-3.5 improves general-response win rate over Generative Agents (36.8 vs 20.7 in the GPT-3.5 comparison table).
- On uncommon scripts, GPT-3.5 with RoleAgent raises self-knowledge accuracy from 55.0% to 77.5% over using GPT-3.5 alone.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add the hierarchical memory system and memory replay rather than using neither (ablation) | RoleAgent with GPT-3.5 · RoleAgentBench English | Self-Knowledge accuracy: 70.3 -> 87.6 | Helped |
| Add the hierarchical memory system and memory replay rather than using neither (ablation) | RoleAgent with GPT-3.5 · RoleAgentBench English | Reaction accuracy: 54.5 -> 76.7 | Helped |
| Add the hierarchical memory system and memory replay rather than using neither (ablation) | RoleAgent with GPT-3.5 · RoleAgentBench English | Summarization WR-G: 29.5 -> 46.7 | Helped |
| Add memory replay (ablation) | RoleAgent with GPT-3.5 · RoleAgentBench English | Self-Knowledge accuracy: 73.6 -> 87.6 | Helped |
| Add memory replay (ablation) | RoleAgent with GPT-3.5 · RoleAgentBench English | Summarization WR-G: 35.4 -> 46.7 | Helped |
| Add hierarchical memory (ablation) | RoleAgent with GPT-3.5 and memory replay · RoleAgentBench English | Reaction accuracy: 54.5 -> 76.4 | Helped |
| Add hierarchical memory (ablation) | RoleAgent with GPT-3.5 and memory replay · RoleAgentBench English | Self-Knowledge accuracy: 70.3 -> 73.6 | Helped |
| Use RoleAgent instead of Generative Agents | GPT-3.5 role-playing agents · RoleAgentBench English | Self-Knowledge accuracy: 68.9 -> 87.6 | Helped |
| Use RoleAgent instead of Generative Agents | GPT-3.5 role-playing agents · RoleAgentBench English | Reaction accuracy: 40.7 -> 76.7 | Helped |
| Use RoleAgent instead of Generative Agents | GPT-3.5 role-playing agents · RoleAgentBench English | General Response WR-G: 31.8 -> 37.6 | Helped |
| Use a stronger API-based base model rather than an open-source model | RoleAgent · RoleAgentBench English | Self-Knowledge accuracy | Helped |
| Use an English-centric LLM on the Chinese subset instead of its English subset | LLaMA3-8B · RoleAgentBench Chinese vs English | Self-Knowledge accuracy: 84.0 -> 70.1 | Hurt |
Surprising findings
- RoleLLM, fine-tuned on role-related data, performs worse than general prompted LLMs on RoleAgentBench, which the authors attribute to its focus on speaking style.
- GPT-4 judge agreement with human evaluators on general response is low (agreement percentages between about 25% and 56%).
Limitations
- Extensive API and GPU costs are needed for building and interacting with RoleAgent (about $5,000 in OpenAI API calls).
- Generated content may reproduce aggressive, obscene, or prejudiced persona traits.
- Benchmark questions and answers are generated by GPT-4 and then revised by annotators, so the benchmark depends on GPT-4.
Experimental setup
Datasets: RoleAgentBench RoleBench 20 English scripts 5 Chinese scripts
Benchmarks: RoleAgentBench
Metrics: accuracy win rate entity density entity recall ER/ED
Baselines: GPT-4 Qwen-Max GPT-3.5 Yi-34B Qwen1.5-14B Mistral-7B LLaMA3-8B ChatGLM3-6B RoleLLM Generative Agents
Builds on: Generative Agents RoleLLM Faiss
Closest papers by meaning
- DMT-RoleBench: A Dynamic Multi-Turn Dialogue Based Benchmark for Role-Playing Evaluation of Large Language Model and Agent (AAAI 2025 · similarity 0.87)
- MMRole: A Comprehensive Framework for Developing and Evaluating Multimodal Role-Playing Agents (ICLR 2025 · similarity 0.87)
- CoSER: Coordinating LLM-Based Persona Simulation of Established Roles (ICML 2025 · similarity 0.85)
- Codifying Character Logic in Role-Playing (NeurIPS 2025 · similarity 0.85)
- ProAgent: Building Proactive Cooperative Agents with Large Language Models (AAAI 2024 · similarity 0.84)
- ProSoftArena: Benchmarking Hierarchical Capabilities of Multi-modal Agents in Professional Software Environments (CVPR 2026 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.