Research map / Large language models / AI and society
Human Simulacra: Benchmarking the Personification of Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds Human Simulacra, a dataset of 11 virtual characters with life stories and Jung-based personality descriptions, plus a self-report/observer-report/conformity evaluation and a multi-agent cognitive memory mechanism (MACM) for LLM personification.
Problem
Existing LLM role-play and human simulation work lacks grounded, psychology-based characters, detailed life stories, and rigorous evaluation for replacing human participants in psychological and sociological experiments.
Approach
Virtual characters built from scratch with detailed human-reviewed life stories and an eight-dimensional Jungian personality model (640 trait descriptions), evaluated via self-reports, third-party observer reports, and an Asch conformity replication, with a Multi-Agent Cognitive Mechanism that converts life stories into long-term memory and coordinates thinking, emotion, and memory agents.
Key results
- MACM with GPT-4-Turbo reaches 88.00 on self-report, versus 78.67 for Prompt and 85.33 for RAG on the same model.
- On observer reports for GPT-4-Turbo, MACM achieves a final score of 77.50 versus 69.00 for Prompt and 65.50 for RAG.
- In the Asch-style conformity experiment, MACM shows submissive reactions similar to humans but with less variability, while Character.ai scores 0% correct on all critical trials.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MACM instead of RAG | GPT-4-Turbo simulacra · Self reports | Sum: 85.33 -> 88.00 | Helped |
| Use MACM instead of Prompt | GPT-4-Turbo simulacra · Observer reports | Final Score: 69.00 -> 77.50 | Helped |
| Use MACM instead of RAG | GPT-4-Turbo simulacra · Observer reports | Final Score: 65.50 -> 77.50 | Helped |
| Add the Thinking Agent (ablation) | MACM on GPT-4 · Self reports | Sum: 81.33 -> 86.67 | Helped |
| Add the Thinking Agent (ablation) | MACM on GPT-4-Turbo · Self reports | Sum: 83.33 -> 88.00 | Helped |
| Add the Thinking Agent (ablation) | MACM on Qwen-turbo · Self reports | Sum: 66.00 -> 74.67 | Helped |
| Add the Emotion Agent (ablation) | MACM on GPT-4 · Self reports | Sum: 83.33 -> 86.67 | Helped |
| Add the Emotion Agent (ablation) | MACM on GPT-4-Turbo · Self reports | Sum: 85.33 -> 88.00 | Helped |
| Add the Emotion Agent (ablation) | MACM on Qwen-turbo · Self reports | Sum: 71.33 -> 74.67 | Helped |
| Add the Memory Agent (ablation) | MACM on GPT-4 · Self reports | Sum: 82.67 -> 86.67 | Helped |
| Add the Memory Agent (ablation) | MACM on GPT-4-Turbo · Self reports | Sum: 84.00 -> 88.00 | Helped |
| Add the Memory Agent (ablation) | MACM on Qwen-turbo · Self reports | Sum: 68.67 -> 74.67 | Helped |
Surprising findings
- Blank models without any character knowledge still score points on single- and multiple-choice self-report questions by random guessing.
- RAG-based simulacra often score lower than Prompt-based ones on weaker LLMs because large amounts of retrieved text interfere with self-positioning.
Limitations
- Observer reports rely on a small panel of human judges and 55 scenarios, and MACM with high-quality results is limited to high-capability models like GPT-4-Turbo.
- Dataset construction is costly, requiring many API calls and at least five days of human review, limiting scale to 11 characters.
- Jung's typology has limited empirical support compared with the Big Five, and nationality and race were deliberately omitted from character attributes.
Experimental setup
Datasets: Human Simulacra
Benchmarks: Human Simulacra self-report questionnaires Human Simulacra observer reports Asch conformity replication
Metrics: exact-match accuracy on cloze, single-choice and multiple-ch description matching score response similarity score ICC average correct rate on critical trials
Baselines: Prompt RAG None (blank model) Character.ai Character-LLM
Builds on: Role-LLM Character-LLM Asch conformity experiment Jung personality typology Multi Store Model of memory
Closest papers by meaning
- OdysSim: Building Foundation Models for Human Behavior Simulation (NeurIPS 2026 · similarity 0.88)
- Position: LLM Social Simulations Are a Promising Research Method (ICML 2025 · similarity 0.87)
- Open Models, Closed Minds? On Agents Capabilities in Mimicking Human Personalities through Open Large Language Models (AAAI 2025 · similarity 0.87)
- Persona Generators: Generating Diverse Synthetic Personas for Arbitrary Contexts (NeurIPS 2026 · similarity 0.87)
- Towards Real-world Human Behavior Simulation: Benchmarking Large Language Models on Long-horizon, Cross-scenario, Heterogeneous Behavior Traces (NeurIPS 2026 · similarity 0.87)
- Self-Alignment of Large Language Models via Monopolylogue-based Social Scene Simulation (ICML 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.