Research map / Large language models / Agents, code and math
Generating Computational Cognitive models using Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes GeCCo, an in-context LLM pipeline that generates, fits, and iteratively refines cognitive models as Python functions, matching or beating literature models across four cognitive domains.
Problem
Handcrafting computational cognitive models is slow and requires domain expertise, and researchers may miss better models outside their own theoretical commitments.
Approach
Prompt open-source LLMs with task description, participant data, a code template and BIC-based feedback over 10 iterations to propose Python cognitive models; fit proposals offline with SciPy on held-out data and keep the best, with no fine-tuning.
Key results
- In the decision-making task, Llama's generated model reached BIC 39.40 vs 51.97 for the pWADD literature model.
- In the working memory task, Llama's model reached BIC 267.25 vs 275.2 for RL-WM, with exceedance probability 0.99 vs 0.01.
- In the learning task, the LLM model had EXP 0.97 vs 0.03 for RW-4alpha despite only slightly lower BIC (77.97 vs 85.24).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Llama-generated GeCCo model instead of pWADD | Decision-making model fit · Decision making | BIC: 51.97 \pm 4.52 vs. 39.40 \pm 4.54 | Helped |
| Use Llama-generated GeCCo model instead of RW^{4\alpha} | Learning model fit · Learning | BIC: 85.24 \pm 7.95 vs. 77.97 \pm 5.40 | Helped |
| Use R1-generated GeCCo model instead of Hybrid | Planning model fit · Planning | BIC: 437.38 \pm 18.72 vs. 432.47 \pm 19.31; p = 0.27 | No clear effect |
| Use Llama-generated GeCCo model instead of RL-WM | Working-memory model fit · Working memory | BIC: 275.2 \pm 9.21 vs. 267.25 \pm 9.31 | Helped |
| Use LLM-generated GeCCo model instead of Centaur | Decision-making model · Decision-making | negative log-likelihood: 15.41 \pm 1.99 vs. 15.08 \pm 2.29; p = 0.78 | No clear effect |
| Use LLM-generated GeCCo model instead of Centaur | Learning model · Learning | negative log-likelihood: 53.58 \pm 3.31 vs. 26.17 \pm 2.79 | Helped |
| Use LLM-generated GeCCo model instead of Centaur | Planning model · Planning | negative log-likelihood: 206.00 \pm 11.14 vs. 214.10 \pm 10.59; p = 0.18 | No clear effect |
| Use LLM-generated GeCCo model instead of Centaur | Memory model · Memory | negative log-likelihood: 127.44 \pm 4.68 vs. 120.67 \pm 4.68 | Helped |
| Add iterative feedback to the prompt (ablation) | GeCCo prompt · Across cognitive domains | BIC: \beta_{M \pm \text{SEM}} = -17.040~\pm~2.287, p < 0.05 | Helped |
| Include participant data sequences in the prompt (ablation) | GeCCo prompt · Across cognitive domains | BIC: \beta_{M \pm \text{SEM}} = -12.836~\pm~2.287, p < 0.05 | Helped |
| Include task description in the prompt (ablation) | GeCCo prompt · Across cognitive domains | BIC: \beta_{M \pm \text{SEM}} = -9.834~\pm~2.287, p < 0.05 | Helped |
| Include code template in the prompt (ablation) | GeCCo prompt · Across cognitive domains | BIC: \beta_{M \pm \text{SEM}} = -9.359~\pm~2.287, p < 0.05 | Helped |
Surprising findings
- Removing the code template had the smallest impact among prompt components in the ablation, while feedback was most important.
- Llama outperformed the larger Qwen2.5-72B as a base model, while reasoning ability (R1) had no significant effect.
Limitations
- Only low-dimensional cognitive tasks were examined; generalization to high-dimensional data such as vision or language is untested.
- Feedback relies on offline BIC computed outside the LLM, with no LLM-based evaluation of plausibility or parsimony.
- Planning-task generated model did not significantly beat the Hybrid model on BIC (p=0.27); advantage there rests on exceedance probability.
Experimental setup
Datasets: multi-feature decision making dataset two-armed bandit dataset with partial and full feedback two-stage planning dataset RL-WM working memory dataset synthetic Take the Best and Tallying data synthetic Rescorla-Wagner data
Benchmarks: decision making human data learning human data planning human data working memory human data synthetic recovery tasks
Metrics: BIC AIC exceedance probability negative log-likelihood posterior predictive checks identification accuracy
Baselines: pWADD RW-4alpha Hybrid model RL-WM model Centaur
Builds on: Rescorla-Wagner model Bayesian Information Criterion Centaur LogProber
Closest papers by meaning
- Turning large language models into cognitive models (ICLR 2024 · similarity 0.93)
- Can Large Language Models Derive High-Level Cognition from Low-Level and Fragmented Foundational Information? (AAAI 2025 · similarity 0.90)
- CogBench: a large language model walks into a psychology lab (ICML 2024 · similarity 0.90)
- Discovering Symbolic Cognitive Models from Human and Animal Behavior (ICML 2025 · similarity 0.90)
- Language Models Trained to do Arithmetic Predict Human Risky and Intertemporal Choice (ICLR 2025 · similarity 0.89)
- Using Artificial Populations to Study Psychological Phenomena in Neural Models (AAAI 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.