Research map / Large language models / Language, safety and interpretability

CharacterBench: Benchmarking Character Customization of Large Language Models

AAAI 2025 · Large language models › Language, safety and interpretability · approach: LLM unlearning and benchmarks · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces CharacterBench, a large bilingual generative benchmark (22,859 human-annotated samples, 3,956 characters, 11 dimensions in 6 aspects) for LLM character customization, plus the CharacterJudge evaluator fine-tuned from Qwen2-7B-Chat.

Problem

Existing character-customization benchmarks cover few character categories and dimensions, and feature sparsity in open-ended responses makes generative evaluation ineffective and inefficient.

Approach

Classify evaluation dimensions as sparse or dense by whether character features reliably appear in each response; craft target-oriented queries (from profile/context fragments) for sparse dimensions and target-free queries for dense ones, then train a judge (CharacterJudge) on human scores to score responses cheaply and stably.

Key results

• CharacterJudge reaches substantially higher Pearson correlation with human scores than GPT-4 and GLM-4 judges, with 42% and 36% average improvements over the suboptimal GLM-4-TG on AVG.
• Benchmark ranking of LLMs correlates with human rankings at Spearman 73.1% overall, versus 21.4% for CharacterEval and 38.1% for SocialBench.
• DPO on CharacterGLM-6B with CharacterJudge-scored pairs wins 42.2% vs 34.3% lose against the vanilla model (+7.9 improvement).

Measured effects reported in the paper

Surprising findings

• Bilingual fine-tuning was found less effective than training each language separately, so separate Chinese and English judges were trained.
• Most LLMs consistently struggle on the fact accuracy dimension, while morality scores are high across all models.

Limitations

• Morality data contains sensitive and unethical content, requiring restricted access.
• Human-annotation-based scores are limited to a 4-point or other coarse scales per dimension.

Experimental setup

Datasets: CharacterEval test set human role-playing corpus human-prototype interaction corpus synthesized dialogues via prototype interaction

Benchmarks: CharacterBench

Metrics: Pearson correlation with human scores Spearman correlation Kendall correlation win/tie/lose rate average dimension score

Baselines: GPT-4 GPT-3.5-turbo GPT-4-1106 GLM-4 GPT-4o CharacterEval SocialBench CharacterGLM-6B

Builds on: CharacterEval SocialBench direct preference optimization self-consistency

Closest papers by meaning

• CoSER: Coordinating LLM-Based Persona Simulation of Established Roles (ICML 2025 · similarity 0.87)
• Codifying Character Logic in Role-Playing (NeurIPS 2025 · similarity 0.86)
• WritingBench: A Comprehensive Benchmark for Generative Writing (NeurIPS 2025 · similarity 0.85)
• WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the Wild (ICLR 2025 · similarity 0.85)
• DMT-RoleBench: A Dynamic Multi-Turn Dialogue Based Benchmark for Role-Playing Evaluation of Large Language Model and Agent (AAAI 2025 · similarity 0.85)
• DRESSing Up LLM: Efficient Stylized Question-Answering via Style Subspace Editing (ICLR 2025 · similarity 0.84)