Research map / Large language models / AI and society
Using Artificial Populations to Study Psychological Phenomena in Neural Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PopulationLM, which builds artificial populations of language models via stratified Monte Carlo dropout, and uses them to re-examine typicality and structural priming effects in BERT, GPT-2, and RoBERTa family models.
Problem
Studies of cognitive behavior in transformer language models typically rely on single models and lack the population-level statistical rigor used in human psychology, so their conclusions may not be robust.
Approach
Applies Monte Carlo dropout at inference to create a fixed population of masked model instances (stratified MC dropout, 50 members) and analyzes their outputs directly as a population with statistical tests (KS, Pearson, Wilcoxon), rather than aggregating them.
Key results
- Population correlation between probability and typicality is lower than single-model base correlation, indicating single models overestimate cognitive effects; within well-represented categories typicality explains up to ~20% of probability variance.
- Masked language models show significant positive correlation between normalized population uncertainty and decreasing typicality, while all causal LMs show negative certainty/typicality correlation.
- Structural priming is largely absent: only GPT-2 and RoBERTa prefer primed over control sentences, and the alternative (non-matching structure) treatment produced an average effect 96% as large as the priming effect.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use population correlations instead of the single base model's total correlation | Language-model typicality analysis · Typicality | probability-typicality correlation | Hurt |
| Increase dropout rate above 0.1 | MC dropout populations · Typicality and structural priming experiments | behavioral correlations: beyond rates of 0.5 | Hurt |
| Use categories well represented in training data | Language-model populations · Typicality | probability-typicality correlation: 10\%<r^2<25\% | Helped |
| Use a population of 50 members rather than a single model | Artificial model populations · Typicality and structural priming experiments | statistical significance: 50 | Helped |
| Prime with a similar sentence structure rather than use the unprimed control | GPT-2 and RoBERTa models · Structural priming | Wilcoxon preference for PT over CT: 0.96 (p$\approx$0); 0.99 (p$\approx$0) | Mixed |
| Attribute the primed response to structural similarity rather than a general priming effect | GPT-2 family · Structural priming | Pearson r between AT-CT and PT-CT: 0.89 (p$<$0.01); 0.93 (p$<$0.01); 0.94 (p$<$0.01) | No clear effect |
| Use similar-structure priming rather than alternative-structure priming | RoBERTa family · Structural priming | structural priming effect distinct from alternative priming: 0.98 $\pm$0.001; 0.96 $\pm$0.001 | No clear effect |
Surprising findings
- Alternative non-priming treatment produced ~96% of the average effect of structural priming, suggesting the apparent priming effect is not structure-specific.
- Causal language models show negative certainty/typicality correlation, opposite to masked language models.
Limitations
- Dropout populations may not approximate human sample variability; the authors do not claim they approximate human studies.
- KS test effect sizes are context-specific and the population method's benefit varies by model and behavior.
- Experiments are limited to older encoder and decoder models (BERT, GPT-2, RoBERTa) and do not test closed-source or large LLMs.
Experimental setup
Datasets: typicality data from prior work BERT family training data frequencies structural priming sentence data (3000 examples)
Benchmarks: typicality ranking data structural priming sentence set
Metrics: Pearson correlation Spearman correlation Kolmogorov-Smirnov statistic Wilcoxon signed-rank test normalized population standard deviation
Baselines: single-model base evaluation
Builds on: Monte Carlo dropout
Closest papers by meaning
- Turning large language models into cognitive models (ICLR 2024 · similarity 0.90)
- CogBench: a large language model walks into a psychology lab (ICML 2024 · similarity 0.89)
- Generating Computational Cognitive models using Large Language Models (NeurIPS 2025 · similarity 0.89)
- Language Models Trained to do Arithmetic Predict Human Risky and Intertemporal Choice (ICLR 2025 · similarity 0.88)
- Language Model Behavioral Phases are Consistent Across Architecture, Training Data, and Scale (NeurIPS 2025 · similarity 0.88)
- Bias Amplification in Language Model Evolution: An Iterated Learning Perspective (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.