Research map / Large language models / Agents, code and math
Open Models, Closed Minds? On Agents Capabilities in Mimicking Human Personalities through Open Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Assesses 12 open LLM agents on MBTI and BFI tests, measuring intrinsic personalities and their ability to mimic personality and role-conditioned personas via AutoGen-based interview simulations.
Problem
Personality of LLMs has been studied mostly on closed commercial models with individual tests; open LLMs' intrinsic personalities and their ability to mimic imposed human personalities or roles are unexplored.
Approach
Treat open LLMs as interviewer/interviewee agents in AutoGen and administer full MBTI (60 questions) and BFI (44 questions) tests under unconditioned, personality-conditioned, and role+personality-conditioned prompts at two temperatures, with many repetitions, to measure intrinsic traits and conditioning fidelity.
Key results
- Open LLM agents at temperature 0.01 predominantly output ENFJ on MBTI; Llama family, Gemma and Phi3 stay at ENFJ regardless of temperature.
- Only SOLAR (0.74-0.79 accuracy), Dolphin (0.63-0.65), NeuralChat (0.50-0.58) and Llama3-8 (0.48-0.52) reliably mimic conditioned MBTI types; most models remain closed-minded.
- Role plus personality conditioning helps mainly SOLAR, NeuralChat, Llama3-8 and Dolphin; Neuroticism conditioning is most effective in BFI (above 230% increase for Llama3-8).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Condition on target MBTI personality | SOLAR · MBTI personality-conditioned test | accuracy: 0.785 ±0.391 (τ = 0.01); 0.744 ±0.353 (τ = 0.7) | Helped |
| Condition on target MBTI personality | Dolphin · MBTI personality-conditioned test | accuracy: 0.654 ±0.459 (τ = 0.01); 0.633 ±0.379 (τ = 0.7) | Helped |
| Condition on target MBTI personality | NeuralChat · MBTI personality-conditioned test | accuracy: 0.577 ±0.479 (τ = 0.01); 0.498 ±0.328 (τ = 0.7) | Helped |
| Condition on target MBTI personality | Llama3-8 · MBTI personality-conditioned test | accuracy: 0.517 ±0.487 (τ = 0.01); 0.479 ±0.408 (τ = 0.7) | Helped |
| Condition on target MBTI personality | Mistral and Mixtral · MBTI personality-conditioned test | accuracy: Mistral: 0.062 ±0.242 (τ = 0.01), 0.062 ±0.169 (τ = 0.7); Mixtral: 0.062 ±0.166 (τ = 0.01), 0.060 ±0.168 (τ = 0.7) | Hurt |
| Condition on target MBTI personality | Llama2-13 · MBTI personality-conditioned test | accuracy: 0.283 ±0.433 (τ = 0.01); 0.265 ±0.348 (τ = 0.7) | Hurt |
| Increase temperature from τ = 0.01 to τ = 0.7 | MBTI personality-conditioned agents · MBTI personality-conditioned test | accuracy | Hurt |
| Condition on Extraversion | SOLAR · BFI | Extraversion score increase (%): 47.4 (τ = 0.01); 41.6 (τ = 0.70) | Helped |
| Condition on Conscientiousness | Dolphin · BFI | Conscientiousness score increase (%): 40.0 (τ = 0.01); 30.3 (τ = 0.70) | Helped |
| Condition on Neuroticism | Llama3-8 · BFI | Neuroticism score increase (%): 233.3 (τ = 0.01); 150.8 (τ = 0.70) | Helped |
| Condition on Neuroticism | Mixtral · BFI | Neuroticism score increase (%): 0.0 (τ = 0.01); -0.4 (τ = 0.70) | No clear effect |
| Condition on Agreeableness | Llama2-7 · BFI | Agreeableness score increase (%): 5.7 (τ = 0.01); -14.0 (τ = 0.70) | Mixed |
Surprising findings
- Mistral and Mixtral largely disregard personality conditioning and keep their inherent type, while the Dolphin variant of Mistral performs much better than its base.
- Falcon and some models show negative side effects where double conditioning decreases the BFI score on the target factor.
Limitations
- Evaluation limited to English and to models with manageable size (up to ~47B); larger Llama/Falcon variants not tested.
- MBTI assessed via 16personalities.com rather than official MBTI test due to cost.
- Limited explainability because training data and methods of the open models are not fully disclosed.
Experimental setup
Datasets: StereoSet profession catalog (120 roles) MBTI questions from 16Personalities (60) BFI questions (44)
Benchmarks: MBTI test BFI test
Metrics: MBTI type accuracy BFI factor score word overlap semantic cosine similarity
Baselines: unconditioned prompting
Builds on: AutoGen MBTI Big Five Inventory
Closest papers by meaning
- CogBench: a large language model walks into a psychology lab (ICML 2024 · similarity 0.88)
- OdysSim: Building Foundation Models for Human Behavior Simulation (NeurIPS 2026 · similarity 0.88)
- Human Simulacra: Benchmarking the Personification of Large Language Models (ICLR 2025 · similarity 0.87)
- ToMATO: Verbalizing the Mental States of Role-Playing LLMs for Benchmarking Theory of Mind (AAAI 2025 · similarity 0.87)
- Position: Theory of Mind Benchmarks are Broken for Large Language Models (ICML 2025 · similarity 0.87)
- Personality Alignment of Large Language Models (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.