Research map / Large language models / Agents, code and math
CoSER: Coordinating LLM-Based Persona Simulation of Established Roles
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces the COSER dataset of authentic multi-character book dialogues, given-circumstance acting for training and evaluating role-playing LLMs, and COSER 8B/70B models fine-tuned from LLaMA-3.1 that match or beat GPT-4o on role-play benchmarks.
Problem
Role-playing language agents for established characters lack authentic character datasets and nuanced evaluation; existing data is LLM-synthesized or limited to two-character dialogues, and LLM judges are biased and shallow.
Approach
Extract authentic multi-character conversations with rich context (settings, motivations, experiences, inner thoughts, actions) from 771 books via an LLM pipeline; train actors to portray each character (given-circumstance acting) and evaluate via multi-agent simulation scored by penalty-based LLM critics with rubrics and original dialogue as reference.
Key results
- COSER 70B reaches 93.47% accuracy on LifeChoice and 75.80% on InCharacter, surpassing GPT-4o on LifeChoice by 23%.
- COSER 70B achieves the best average GCA score among open models and is competitive with GPT-4o; BLEU exceeds the second-best model by 58%.
- In human evaluation, COSER 70B obtains the highest score (6.783) and win rate (86.9%) among 7 models.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune LLaMA-3.1-70B on authentic COSER data | LLaMA-3.1-70B · COSER Test | Average score: 55.49±0.33 -> 59.06±0.22 | Helped |
| Fine-tune LLaMA-3.1-8B on authentic COSER data | LLaMA-3.1-8B · COSER Test | Average score: 52.99±1.20 -> 56.45±0.56 | Helped |
| Fine-tune LLaMA-3.1-70B on authentic COSER dialogues | LLaMA-3.1-70B · COSER Test | BLEU: 4.82±0.06 -> 10.10±0.04 | Helped |
| Include inner thoughts in test-time input (ablation) | COSER-70B · COSER Test | Average score: 57.32 -> 59.06 | Helped |
| Include motivations in test-time input (ablation) | COSER-70B · COSER Test | Average score: 57.71 -> 59.06 | Helped |
| Train with inner thoughts (ablation) | COSER-70B · COSER Test | Average score: 56.04 -> 59.06 | Helped |
| Include inner thoughts in test-time input (ablation) | COSER-8B · COSER Test | Average score: 54.65 -> 56.45 | Helped |
| Include motivations in test-time input (ablation) | COSER-8B · COSER Test | Average score: 56.81 -> 56.45 | Hurt |
| Train with inner thoughts (ablation) | COSER-8B · COSER Test | Average score: 54.25 -> 56.45 | Helped |
| Use original dialogues as references in GCA judging (ablation) | GCA (GPT-4o) · Human evaluation on COSER Test | Alignment: 64.3 -> 68.6 | Helped |
| Use scoring rubrics in GCA judging (ablation) | GCA (GPT-4o) · Human evaluation on COSER Test | Alignment: 65.1 -> 68.6 | Helped |
| Apply length correction in GCA judging (ablation) | GCA (GPT-4o) · Human evaluation on COSER Test | Alignment: 64.5 -> 68.6 | Helped |
Surprising findings
- Higgs-Llama-3-70B, fine-tuned on synthesized dialogues, performs below its LLaMA-3.1 70B base, suggesting authentic data matters more.
- Raw text retrieval barely improves role-play performance, while retrieved experiences and conversations help.
Limitations
- Dataset built with Claude-3.5-Sonnet and evaluated with GPT-4o critics, so pipeline and judging depend on closed LLMs.
- Dataset is for research only and raw novel text is not distributed due to copyright, limiting reproducibility.
- Human evaluation is small (60 samples, 3 annotators) and manual evaluation is very time-consuming.
Experimental setup
Datasets: COSER dataset Tulu-3
Benchmarks: COSER Test InCharacter LifeChoice CroSS
Metrics: penalty-based LLM judge score BLEU ROUGE-L accuracy human score win rate alignment with human judges
Baselines: GPT-4o GPT-3.5 Claude-3.5-Sonnet Gemini-1.5-Pro LLaMA-3.1-70B LLaMA-3.1-8B Qwen-2-72B DeepSeek-V3 Higgs-Llama-3-70B
Builds on: LLaMA-3.1 Tulu-3 RoleBench InCharacter given-circumstance acting (Stanislavski)
Closest papers by meaning
- CharacterBench: Benchmarking Character Customization of Large Language Models (AAAI 2025 · similarity 0.87)
- Thinking in Character: Advancing Role-Playing Agents with Role-Aware Reasoning (NeurIPS 2025 · similarity 0.86)
- Codifying Character Logic in Role-Playing (NeurIPS 2025 · similarity 0.86)
- MMRole: A Comprehensive Framework for Developing and Evaluating Multimodal Role-Playing Agents (ICLR 2025 · similarity 0.85)
- Consistently Simulating Human Personas with Multi-Turn Reinforcement Learning (NeurIPS 2025 · similarity 0.85)
- NarrativePlay: An Automated System for Crafting Visual Worlds in Novels for Role-Playing (AAAI 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.