Research map / Large language models / Agents, code and math
STEER: Assessing the Economic Rationality of Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes STEER, a taxonomy-driven MCQA benchmark of economic rationality elements with generated questions and "report card" scoring, and evaluates 14 LLMs.
Problem
Assessing whether LLM agents make economically rational decisions requires a holistic, decomposed methodology rather than ad hoc narrow tasks.
Approach
Derive a taxonomy of 64 elements of economic rationality from economic theory, generate grade-leveled, domain-varied MCQA questions per element with GPT-4 Turbo, and score models via normalized accuracy, calibration, and robustness metrics combined into user-tunable STEER report cards.
Key results
- GPT-4 Turbo was the best model with normalized accuracy 0.3302 and 63.61% exact-match over the whole benchmark, closing only about a third of the gap to perfect answers.
- Models below roughly 40B parameters were not reliably better than random guessing; most 7B and 13B models had negative normalized accuracy.
- GPT-4 Turbo fell to roughly random guessing from grade 9 onward; self-explanation improved performance for all models and few-shot prompting helped up to three examples.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Model size increased (larger models) | open-source LLMs · STEER benchmark | normalized accuracy | Helped |
| Self-explanation (separate or together prompting) (ablation) | all models · STEER benchmark | accuracy | Helped |
| Few-shot prompting up to 3 examples (ablation) | all models · STEER benchmark | accuracy | Helped |
| Few-shot prompting beyond 3 examples (4 or more) (ablation) | all models · STEER benchmark | accuracy | Hurt |
| Self-explanation (ablation) | GPT-4 Turbo, Avoidance of Certainty Effect · STEER benchmark | normalized accuracy: 0.079 -> 0.508 | Helped |
| Self-explanation (ablation) | GPT-4 Turbo, Avoidance of Reflection Effect · STEER benchmark | normalized accuracy: 0.390 -> 0.865 | Helped |
| Increasing grade level (harder questions) | all models · STEER grade-specific SRCs | normalized accuracy | Hurt |
| Iterated Removal of Dominated Strategies is harder than its prerequisite-level Pure Nash Equilibrium task | LLMs across the board · Dependency robustness | normalized accuracy | Hurt |
Surprising findings
- GPT-4 Turbo performed worse than random guessing on Best Response, and performed worse on Iterated Removal of Dominated Strategies than on its descendant Pure Nash Equilibrium.
- Few-shot prompting performance decreased beyond three examples.
Limitations
- Results depend on the chosen prompting and adaptation strategies; performance may be higher with better prompts, so reported numbers are lower bounds.
- No global ranking of models is possible since users choose their own report card rubric and aggregation.
- Only 49 of 64 elements have written templates, and tests cover single-element questions rather than combinations.
Experimental setup
Datasets: STEER benchmark (synthetic generated questions)
Benchmarks: STEER
Metrics: normalized accuracy exact-match accuracy expected calibration error domain robustness dependency robustness
Baselines: GPT-3.5 Turbo Llama-2 70b Llama 65b Falcon 40b
Builds on: von Neumann-Morgenstern expected utility MMLU BIG-bench Hard HellaSwag
Closest papers by meaning
- STEER-ME: Assessing the Microeconomic Reasoning of Large Language Models (NeurIPS 2025 · similarity 0.90)
- It Helps to Take a Second Opinion: Teaching Smaller LLMs To Deliberate Mutually via Selective Rationale Optimisation (ICLR 2025 · similarity 0.88)
- Language Models Trained to do Arithmetic Predict Human Risky and Intertemporal Choice (ICLR 2025 · similarity 0.88)
- LLM Agents Can Be Choice-Supportive Biased Evaluators: An Empirical Study (AAAI 2025 · similarity 0.88)
- Decision-Making Behavior Evaluation Framework for LLMs under Uncertain Context (NeurIPS 2024 · similarity 0.87)
- Distributive Fairness in Large Language Models: Evaluating Alignment with Human Values (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.