Research map / Large language models / Agents, code and math
Codifying Character Logic in Role-Playing
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Codified Profiles, which compile textual character profiles into executable parse_by_scene functions with LLM-based condition checks, and a Fandom-based benchmark (83 characters, 5,141 scenes) showing gains in persistence, updatability, and controllable randomness, especially for small models.
Problem
Prompt-based role-playing appends character profiles to prompts, so character behavior depends on the LLM's implicit reasoning, which is inconsistent, hard to debug or update, and computationally costly at runtime.
Approach
Use an LLM to codify each profile paragraph into an executable function whose control flow (if-then-else) is evaluated deterministically, with semantic conditions checked by the role-playing LLM via logits over yes/no/unknown verbalizers, and randomness encoded directly in code.
Key results
- Codified Profile averages 68.38 NLI on main characters vs 65.98 for original textual profile and 64.21 for no-profile vanilla (llama-3.1-8b), and 69.87 vs 65.70 on minor characters.
- A 1B model with codified profile and a distilled 0.1B deberta condition checker reaches 60.21 on main characters vs 65.98 for 8B with textual profile.
- Distilled deberta-v3-base condition checker reaches 70.53% consistency with gpt-4.1 on held-out data.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace textual profiles with codified profiles | Llama-3.1-8B-Instruct role-playing · Fandom Benchmark (main characters, average) | NLI score: 65.98 -> 68.38 | Helped |
| Replace textual profiles with codified profiles | Llama-3.1-8B-Instruct role-playing · Fandom Benchmark (minor characters, average) | NLI score: 65.70 -> 69.87 | Helped |
| Replace Codified RAG with structured executable behavior logic | Llama-3.1-8B-Instruct role-playing · Fandom Benchmark (main characters, average) | NLI score: 66.04 -> 68.38 | Helped |
| Add a textual profile on top of the codified profile (ablation) | Codified Profile · Fandom Benchmark (main characters, average) | NLI score: 68.38 -> 68.62 | Helped |
| Add a textual profile on top of the codified profile (ablation) | Codified Profile · Fandom Benchmark (minor characters, average) | NLI score: 69.87 -> 69.96 | Helped |
| Add codified profiles instead of textual profiles | Llama-3.2-1B role-playing · Fandom Benchmark (main characters, average) | NLI score: 56.00 -> 58.43 | Helped |
| Add codified profiles instead of textual profiles | Llama-3.2-1B role-playing · Fandom Benchmark (minor characters, average) | NLI score: 59.40 -> 62.59 | Helped |
| Add a distilled 100M condition checker to 1B + Code (ablation) | Llama-3.2-1B role-playing · Fandom Benchmark (main characters, average) | NLI score: 58.43 -> 60.21 | Helped |
| Add a distilled 100M condition checker to 1B + Code (ablation) | Llama-3.2-1B role-playing · Fandom Benchmark (minor characters, average) | NLI score: 62.59 -> 64.25 | Helped |
| Use paragraph-level rather than section-level profile segmentation (ablation) | Codified Profiles · Fandom Benchmark role-playing | NLI score | Helped |
| Use sentence-level rather than paragraph-level profile segmentation (ablation) | Codified Profiles · Fandom Benchmark role-playing | NLI score | Mixed |
| Add targeted reasoning to codified profiles (ablation) | Codified Profile · Fandom Benchmark role-playing | NLI score | Helped |
Surprising findings
- Long chain-of-thought reasoning underperforms, with overthinking derailing character-consistent behavior.
- Textual profiles with temperature sampling show significant bias in simulating precise probabilities, whereas codified randomness controls them precisely.
Limitations
- Codification may lose information from nuanced textual descriptions.
- Only the check_condition helper is explored; no richer operators or world-level codification.
- NLI scoring by an LLM judge (gpt-4.1) may be affected by memorization of famous scenes and cannot capture fine-grained quality differences.
Experimental setup
Datasets: Fandom character profiles and storyline summaries (Haruhi, K-On!, JOJO, FMA, AGOT, ATLA)
Benchmarks: Fandom Benchmark (83 characters, 5,141 scenes)
Metrics: NLI score (entailed=100, neutral=50, contradicted=0) Best@K win rate forward pass count codification precision/recall
Baselines: Vanilla (No Profile) Original Textual Profile Codified RAG Chain-of-thought with textual profile
Builds on: chain-of-thought retrieval-augmented generation natural language inference evaluation
Closest papers by meaning
- Thinking in Character: Advancing Role-Playing Agents with Role-Aware Reasoning (NeurIPS 2025 · similarity 0.88)
- Concept Incongruence: An Exploration of Time and Death in Role Playing (NeurIPS 2025 · similarity 0.87)
- Quantifying and Optimizing Global Faithfulness in Persona-driven Role-playing (NeurIPS 2024 · similarity 0.87)
- CharacterBench: Benchmarking Character Customization of Large Language Models (AAAI 2025 · similarity 0.86)
- CoSER: Coordinating LLM-Based Persona Simulation of Established Roles (ICML 2025 · similarity 0.86)
- ToMATO: Verbalizing the Mental States of Role-Playing LLMs for Benchmarking Theory of Mind (AAAI 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.