Research map / Large language models / Language, safety and interpretability
Persistent Instability in LLM’s Personality Measurements: Effects of Scale, Reasoning, and Conversation History
Tommaso Tosato, Saskia Helbling, Yorguin-Jose Mantilla-Ramos, Mahmood Hegazy, Alberto Tosato, David John Lemay, Irina Rish, Guillaume Dumas
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| increase model size (ablation) | assistant persona, 8 model families · BFI+SD3 personality questionnaires | question-level SD: Spearman p<0.001, negative correlation | Improved |
| increase model size (ablation) | assistant persona · BFI+SD3 personality questionnaires | question-level SD (stability): SD still >0.3 on 5-point scales for 400B+ | Mixed |
| enable chain-of-thought reasoning (ablation) | Qwen3, DeepSeek, Claude, GPT-OSS · BFI+SD3 personality questionnaires | question-level SD: higher in reasoning models (all p<0.01) | Hurt |
| enable chain-of-thought reasoning (ablation) | Qwen3, GPT-OSS · BFI+SD3 personality questionnaires | question-level perplexity: decreased with reasoning (p<0.001) | Improved |
| use LLM-adapted questionnaires instead of original (ablation) | assistant persona · BFI and SD3 | question-level SD: p = 0.286 | No clear effect |
| paraphrase questions instead of reordering them (ablation) | models >=50B · BFI+SD3 | question-level SD: p<0.01 | Hurt |
| paraphrase questions instead of reordering them (ablation) | models <50B · BFI+SD3 | question-level SD: p = 0.244 | No clear effect |
| add conversation history (ablation) | models <50B · BFI+SD3 | question-level SD: p<0.001, increased | Hurt |
| add conversation history (ablation) | models >=50B · BFI+SD3 | question-level SD: p<0.001, decreased | Improved |
| use buddhist persona prompt vs assistant baseline (ablation) | persona prompting · BFI+SD3 | response variability: p<0.05 | Improved |
| use schizophrenia persona prompt vs assistant baseline (ablation) | persona prompting · BFI+SD3 | response variability: p<0.05 | Hurt |
Closest papers by meaning
- Open Models, Closed Minds? On Agents Capabilities in Mimicking Human Personalities through Open Large Language Models (AAAI 2025 · similarity 0.88)
- Rethinking Psychometric Evaluation of LLMs: When and Why Self-Reports Predict Behavior (NeurIPS 2026 · similarity 0.88)
- Bias Runs Deep: Implicit Reasoning Biases in Persona-Assigned LLMs (ICLR 2024 · similarity 0.88)
- Personality Alignment of Large Language Models (ICLR 2025 · similarity 0.88)
- Persona-Model Collapse in Emergent Misalignment (NeurIPS 2026 · similarity 0.87)
- LLM Generated Persona is a Promise with a Catch (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
