Research map / Large language models / AI and society
Language Models Trained to do Arithmetic Predict Human Risky and Intertemporal Choice
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Pretrains a small randomly initialized GPT (Arithmetic-GPT, ~10M params) on synthetic expected-value arithmetic with ecologically distributed probabilities and values, then uses frozen embeddings to predict human risky and intertemporal choices better than classical behavioral models and LLaMA3-70B embeddings.
Problem
Whether LLMs can serve as cognitive models of human decision-making is confounded by huge training data, possible contamination with human data, and value alignment; the origin of human-like choice behavior in LLMs is unclear.
Approach
Instead of using off-the-shelf LLMs, pretrain a small transformer only on the computationally equivalent arithmetic of expected value/present value calculations, with ecologically realistic distributions of probabilities (Beta(0.27,0.27)) and values (power law), and use its frozen embeddings to model human choices.
Key results
- Arithmetic-GPT pretrained on ecological synthetic data reaches R^2 of 70.8% on choices13k and 65.5% on cpc18, outperforming prospect theory (51.5%, 54.2%) and LLaMA3 text embeddings on cpc18 (55.7%).
- On intertemporal choice, Arithmetic-GPT (ecological) gets 67.8% on gershman20 and 95.5% on agrawal23, comparable to LLaMA3 embeddings (66.5%, 98.0%).
- Ablating the answers (random signs) drops R^2 to 33.8% on cpc18, and an untrained model gets 21.0% on choices13k, showing the training data distribution matters.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Pretrain on ecological rather than uniform probability and value distributions | Arithmetic-GPT 10M · choices13k | R²: 69.3% -> 70.8% | Helped |
| Pretrain on ecological rather than uniform probability and value distributions | Arithmetic-GPT 10M · cpc18 | R²: 63.2% -> 65.5% | Helped |
| Pretrain on ecological rather than uniform probability and value distributions | Arithmetic-GPT 10M · gershman20 | R²: 64.0% -> 67.8% | Helped |
| Pretrain on ecological rather than uniform probability and value distributions | Arithmetic-GPT 10M · agrawal23 | R²: 96.1% -> 95.5% | Hurt |
| Add correct equation-answer signs instead of randomizing the signs (ablation) | Arithmetic-GPT with ecological data · Human choice datasets | R²: 61.4% -> 70.8%; 33.8% -> 65.5%; 60.7% -> 67.8%; 80.1% -> 95.5% | Helped |
| Add correct equation-answer signs instead of randomizing the signs (ablation) | Arithmetic-GPT with uniform data · Human choice datasets | R²: 57.5% -> 69.3%; 37.9% -> 63.2%; 59.8% -> 64.0%; 81.1% -> 96.1% | Helped |
| Add arithmetic pretraining (ablation) | Arithmetic-GPT 10M · Human choice datasets | R²: 21.0% -> 70.8%; 28.4% -> 65.5%; 10.8% -> 67.8%; 21.1% -> 95.5% | Helped |
| Reduce ecological pretraining data from 1M to 10K equations (ablation) | Arithmetic-GPT 10M · choices13k and agrawal23 | R²: 57.7 (0.4) / 83.7 (0.3) -> 57.7 (0.5) / 83.6 (0.3) | No clear effect |
| Reduce model size from 10M to 1M parameters (ablation) | Arithmetic-GPT · choices13k and agrawal23 | R²: 57.7 (0.4) / 83.7 (0.3) -> 54.7 (0.6) / 83.4 (0.3) | Mixed |
| Reduce model size from 10M to 30K parameters (ablation) | Arithmetic-GPT · choices13k and agrawal23 | R²: 57.7 (0.4) / 83.7 (0.3) -> 30.6 (0.4) / 56.8 (0.4) | Hurt |
| Change the ecological value power-law exponent from -0.945 to -2 (ablation) | Arithmetic-GPT 10M · choices13k | R²: 57.7 (0.4) -> 55.7 (0.3) | Hurt |
| Change the ecological value power-law exponent from -0.945 to -2 (ablation) | Arithmetic-GPT 10M · agrawal23 | R²: 83.7 (0.3) -> 85.9 (0.3) | Helped |
Surprising findings
- A 10M-parameter model trained only on arithmetic outperforms LLaMA3-70B-Instruct embeddings on risky choice datasets (choices13k, cpc18), and LLaMA3 log-probabilities perform poorly (14.2%, 8.3%).
- Reducing the arithmetic training set to 10K equations barely harms fit to human choices (choices13k and agrawal23 R^2 remain similar).
Limitations
- Requires 1M synthetic equations, far more than human learners would experience, so the data gap is not closed.
- Explained variance is small on some uniform-data variants and the model is limited to autoregressive decoder-only transformers.
- Arithmetic-GPT does not directly model human cognitive processes; it is limited to risky and intertemporal choice and to a restricted numeric range.
Experimental setup
Datasets: choices13k cpc18 gershman20 agrawal23 synthetic arithmetic equations (generated)
Benchmarks: choices13k cpc18 gershman20 agrawal23
Metrics: R^2 (adjusted for logistic regression, squared Pearson r oth top-1 arithmetic accuracy / correct probability
Baselines: Prospect theory Hyperbolic discounting Expected value logistic regression MLP trained on human choices LLaMA3-70B-Instruct
Builds on: GPT prospect theory hyperbolic discounting Centaur
Closest papers by meaning
- Turning large language models into cognitive models (ICLR 2024 · similarity 0.92)
- Large Language Models Assume People are More Rational than We Really are (ICLR 2025 · similarity 0.90)
- CogBench: a large language model walks into a psychology lab (ICML 2024 · similarity 0.90)
- Generating Computational Cognitive models using Large Language Models (NeurIPS 2025 · similarity 0.89)
- Do Language Models Exhibit the Same Cognitive Biases in Problem Solving as Human Learners? (ICML 2024 · similarity 0.89)
- Decision-Making Behavior Evaluation Framework for LLMs under Uncertain Context (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.