Research map / Large language models / Instruction and fine-tuning
CogBench: a large language model walks into a psychology lab
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces CogBench, a text-based benchmark of seven cognitive psychology experiments yielding ten behavioral metrics, and uses it to phenotype 35 LLMs with multilevel regression and prompt-engineering analyses.
Problem
LLM benchmarks focus almost exclusively on performance scores and overlook underlying behavioral mechanisms, so there is no holistic behavioral evaluation of LLMs.
Approach
Adapt canonical cognitive psychology experiments (probabilistic reasoning, horizon task, restless bandit, instrumental learning, two-step task, temporal discounting, BART) into text prompts and derive behavioral metrics (e.g., model-basedness, meta-cognition, directed/random exploration, optimism bias) alongside performance, then relate them to LLM features with multilevel regression.
Key results
- RLHF models were roughly 2x more human-like in UMAP space and showed an 11.7% average decrease in L2-norm distance to humans.
- Parameter count significantly predicted performance (beta=0.277, p<0.001) and model-basedness (beta=0.481, p<0.001).
- Chain-of-thought prompting raised posterior accuracy by 9.01% on average and model-basedness by 64.59%, while take-a-step-back raised model-basedness by 118.59%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add chain-of-thought prompting | Five evaluated LLMs · Probabilistic reasoning | posterior accuracy: $9.01\%$ average increase | Helped |
| Add take-a-step-back prompting | Five evaluated LLMs · Probabilistic reasoning | posterior accuracy: $3.10\%$ increase | Helped |
| Add chain-of-thought prompting | Five evaluated LLMs · Two-step task | model-basedness: $64.59\%$ increase | Helped |
| Add take-a-step-back prompting | Five evaluated LLMs · Two-step task | model-basedness: $118.59\%$ increase | Helped |
| Use RLHF | LLMs · CogBench behavioral metrics | L2-Norm distance from humans: $11.7\%$ average decrease | Helped |
| Increase number of parameters | LLMs · Performance across all seven tasks | standardized performance: $\beta=0.277\pm0.39,z=14.1,p<0.001$ | Helped |
| Increase number of parameters | LLMs · Two-step task | model-basedness: $\beta=0.481\pm0.22,z=4.2,p<0.001$ | Helped |
| Use RLHF | LLMs · Restless bandit task | meta-cognition: $\beta=0.461\pm0.15,z=5.9,p<0.001$ | Helped |
| Fine-tune on code | LLMs · Performance across all seven tasks | standardized performance | No clear effect |
| Fine-tune on code | LLMs · Two-step task | model-basedness | No clear effect |
| Use an open-source model | LLMs · Balloon Analog Risk Task | risk-taking: $\beta=-0.612 \pm 0.11,z=-11.4,p<0.001$ | Hurt |
Surprising findings
- Open-source models were less risk-prone than proprietary models, contrary to the hypothesis that hidden pre-prompts make proprietary models safer.
- Fine-tuning on code showed no substantial effect on performance or model-basedness.
Limitations
- Model features such as parameter counts and training data sizes for proprietary models are uncertain and may be inaccurate, affecting regression precision.
- LLMs were run with reduced trial counts (e.g., about 80 trials in restless bandit versus 400 for humans) due to context limits.
- External validity of the tasks for LLMs has not been established.
Experimental setup
Datasets: CogBench procedurally generated experiment prompts human behavioral data from prior cognitive psychology studies
Benchmarks: CogBench
Metrics: posterior accuracy average delivered dollars model-basedness meta-cognition (adjusted QSR) learning rate optimism bias directed exploration random exploration temporal discounting BART risk-taking
Baselines: text-davinci-003 text-davinci-002 Claude-1 Claude-2 text-bison@002 Falcon-40b MPT-30b LLaMA-2-70
Builds on: chain-of-thought prompting take-a-step-back prompting Rescorla-Wagner model UMAP
Closest papers by meaning
- Turning large language models into cognitive models (ICLR 2024 · similarity 0.91)
- Large Language Models Think Too Fast To Explore Effectively (NeurIPS 2025 · similarity 0.90)
- Scaling Prompt Synthesis for Large Language Model Reasoning (ICML 2026 · similarity 0.90)
- Generating Computational Cognitive models using Large Language Models (NeurIPS 2025 · similarity 0.90)
- Language Models Trained to do Arithmetic Predict Human Risky and Intertemporal Choice (ICLR 2025 · similarity 0.90)
- MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedback (ICLR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.