Research map / Large language models / Alignment and preferences
When Can Digital Personas Reliably Approximate Human Survey Findings?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Evaluates LLM-based digital personas built from LISS panel respondents' background variables and pre-2023 survey histories against their held-out post-2023 answers, across four persona architectures, three LLMs and six reliability dimensions.
Problem
When can LLM digital personas reliably approximate human survey findings, given that plausible-looking synthetic answers may not match the individuals they represent in distributions, individual answers, equity, or multivariate structure.
Approach
A ground-truth same-person temporal holdout design on the LISS panel that tests personas (background-only, profile, profile with lexical or semantic retrieval of prior answers) at question, respondent, distributional, equity, and clustering levels, rather than a single accuracy score.
Key results
- Retrieval-augmented profile (semantic, GPT) reduced question distribution JSD from 0.530 (baseline) to 0.306 and respondent MMD from 0.343 to 0.155.
- Respondent exact match improved modestly from 0.478 (baseline) to 0.531 (profile + semantic, GPT) on core prediction, and clustering ARI stayed near zero (max 0.133).
- Behavioral features (answer variability, answer category count) dominate XGBoost predictions of persona accuracy; XGBoost reached 0.757 accuracy with behavioral features on 132,887 observations.
Surprising findings
- Clustering agreement between persona and human respondent structure is near zero across nearly all settings, even when distributional alignment improves.
- Model choice contributes little; architecture and response-space structure matter more than which LLM is used.
Limitations
- Single longitudinal panel of Dutch households; generalization to other countries and languages untested.
- Restricted to closed-ended questions with finite answer spaces; respondent samples capped at 500 per task.
- Persona profiles generated by a single model (GPT-5.4); profile-generator sensitivity not studied.
Experimental setup
Datasets: LISS panel
Benchmarks: LISS panel held-out 2023-2024 answers
Metrics: weighted F1 exact match rate JSD MMD DPI equity deviation ARI
Baselines: no-context LLM baseline background-only persona
Builds on: LLM survey simulation
Closest papers by meaning
- LLM Generated Persona is a Promise with a Catch (NeurIPS 2025 · similarity 0.89)
- Bias Runs Deep: Implicit Reasoning Biases in Persona-Assigned LLMs (ICLR 2024 · similarity 0.86)
- Persona-Model Collapse in Emergent Misalignment (NeurIPS 2026 · similarity 0.84)
- Uncertainty Quantification for LLM-Based Survey Simulations (ICML 2025 · similarity 0.84)
- Personality Alignment of Large Language Models (ICLR 2025 · similarity 0.83)
- Who's asking? User personas and the mechanics of latent misalignment (NeurIPS 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.