Research map / Large language models / AI and society
Position: LLM Social Simulations Are a Promising Research Method
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Position paper arguing LLM social simulations can become a valid research method by addressing five challenges (diversity, bias, sycophancy, alienness, generalization) with prompting, steering, token sampling, fine-tuning, and iterative evaluation.
Problem
LLM simulations of human research subjects have produced limited and unverifiable results, and few social scientists have adopted them; the paper asks how such simulations could become accurate and verifiable.
Approach
Frames LLM social simulation around five tractable challenges, reviews 53 empirical studies comparing LLM and human data, and proposes concrete directions per challenge (context-rich interview prompting, distribution elicitation, LLM-as-an-expert prompts, steering vectors, temperature variation, fine-tuning on social science data, and iterative OOD evaluation/preregistration).
Key results
- Review of 53 empirical papers comparing LLM simulations to human subjects data, summarized in an appendix table.
- Reports GPT-4 predicted 91% of variation in average treatment effects across 70 preregistered experiments (as cited from prior work).
- Reports Centaur (fine-tuned Llama-3.1-70B on 160 experiments) outperforms existing cognitive models, per cited work.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune Llama-3.1-70B-Instruct on data from 160 human experiments | Llama-3.1-70B-Instruct (Centaur) · Human behavior tasks | simulation accuracy | Helped |
| Add two-hour participant interview transcripts to prompts | GPT-4o participant simulations · Social science studies | maximum predictive-accuracy disparity between demographics | Helped |
| Use verbalization to elicit answer-choice proportions rather than log-probabilities | LLM distribution elicitation · Human response distributions | performance | Mixed |
| Directly elicit distributional data | LLM social simulations · Distributional prediction | accuracy | Hurt |
| Use steering vectors | LLM social simulations · Debiasing and out-of-distribution generalization | usefulness | Hurt |
| Use base models instead of instruction-tuned models | LLM social simulations · Human behavior simulation | simulation performance | Mixed |
| Predict with GPT-4 rather than laypeople | Survey-based experiment forecasting · Novel survey-based experiments | adjusted correlation with average treatment effect: 84% -> 91% | Helped |
| Predict with GPT-3.5 rather than laypeople | Survey-based experiment forecasting · Novel survey-based experiments | adjusted correlation with average treatment effect: 84% -> 82% | Hurt |
Surprising findings
- Sycophancy is a direct consequence of instruction-tuning and may be harder to overcome than effects on diversity and bias.
- Base models or simulations asked to predict as an expert rather than roleplay a persona may reduce sycophancy.
Limitations
- Argument rests on a literature review and commentary rather than new experiments.
- Reviewed studies often lack clear ground truth and rarely test out-of-distribution predictions against human forecasts.
Experimental setup
Datasets: Psych-101 SubPOP Twin-2K-500 World Values Survey American National Election Studies Pew American Trends Panel General Social Survey
Metrics: correlation with treatment effects variance explained accuracy relative to test-retest ceiling
Builds on: Centaur Psych-101
Closest papers by meaning
- MF-LLM: Simulating Population Decision Dynamics via a Mean-Field Large Language Model Framework (NeurIPS 2025 · similarity 0.89)
- OdysSim: Building Foundation Models for Human Behavior Simulation (NeurIPS 2026 · similarity 0.89)
- Towards Real-world Human Behavior Simulation: Benchmarking Large Language Models on Long-horizon, Cross-scenario, Heterogeneous Behavior Traces (NeurIPS 2026 · similarity 0.88)
- Can LLMs Reliably Simulate Human Learner Actions? A Simulation Authoring Framework for Open-Ended Learning Environments (AAAI 2025 · similarity 0.88)
- Human Simulacra: Benchmarking the Personification of Large Language Models (ICLR 2025 · similarity 0.87)
- Can Large Language Model Agents Simulate Human Trust Behavior? (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.