Research map / Large language models / Agents, code and math
STEER-ME: Assessing the Microeconomic Reasoning of Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds STEER-ME, a microeconomics benchmark of 58 non-strategic reasoning elements generated via auto-STEER (LLM style-transfer of templates), and evaluates 27 LLMs with adaptations and calibration metrics.
Problem
Existing LLM benchmarks are narrow and do not comprehensively assess economic reasoning, particularly non-strategic microeconomic reasoning, and static benchmarks risk data contamination.
Approach
Extends the STEER strategic-reasoning benchmark blueprint to non-strategic microeconomics via a taxonomy of 58 elements and auto-STEER, an LLM-assisted template generation pipeline that style-transfers hand-written templates across domains and perspectives and fills numeric fields programmatically to generate fresh questions.
Key results
- o1-preview was the most accurate model on all 13 elements tested, achieving high accuracy even where other models were near random on profit maximization.
- Across the 27 LLMs evaluated, 30 instantiated elements with 5,000-20,000 MCQA questions each; none of the closed models except o1-preview consistently computed deadweight loss.
- gpt-4o's exact-match accuracy on profit maximization dropped from 33.2% (shown) to 27.1% (hidden), near random guessing, indicating reliance on option-plugging.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| show candidate options during reasoning instead of hiding them | gpt-4o on Profit Maximization · Profit Maximization | Exact-match accuracy: \num{27.1}\% -> \num{33.2}\% | Helped |
| increase correct-answer length from 5 digits to 6 digits | claude-3-5-sonnet on Aggregation of Consumer Demand · Aggregation of Consumer Demand | Exact-match accuracy: \num{38}\% -> \num{72}\% | Helped |
| use consumer-goods context instead of technology context | claude-3-5-sonnet on welfare-theorem elements · Welfare theorems | Accuracy: 0.657 | Helped |
| use consumer-goods context instead of technology context | claude-3-haiku on welfare-theorem elements · Welfare theorems | Accuracy: 0.48 | Helped |
| use consumer-goods context instead of technology context | gpt-4o-mini on welfare-theorem elements · Welfare theorems | Accuracy: 0.278 | Helped |
| use non-linear functions rather than Cobb-Douglas functions | LLMs on economic reasoning tasks · Function-family robustness | Performance | Mixed |
| use shown rather than hidden 0-CoT | LLMs on STEER-ME · STEER-ME benchmark | Performance | Helped |
| replace a candidate option with “No other option is correct” when that is the correct answer | claude-3-5-sonnet on Profit Maximization · Profit Maximization | Exact-match accuracy: -\num{20.5}\% | Hurt |
| replace a candidate option with “No other option is correct” when that is the correct answer | o1-preview on Profit Maximization · Profit Maximization | Exact-match accuracy: -\num{15.1}\% | Hurt |
| replace a candidate option with “No other option is correct” when that is the correct answer | gpt-4o on Profit Maximization · Profit Maximization | Exact-match accuracy: -\num{11.2}\% | Hurt |
| use free-text QA instead of multiple-choice options | LLMs on Profit Maximization · Profit Maximization | Exact-match accuracy: no model exceeded \num{10}\% | Hurt |
| score basic exponent-operation ability | LLMs on downstream exponent-based versus linear tasks · Cobb-Douglas downstream tasks | Correlation with reduced performance gap: \num{0.774} ($p = \num{1.23e-07}$) | Helped |
Surprising findings
- Many models solved profit maximization by plugging in candidate answer options rather than deriving the optimum analytically.
- Even large models such as GPT-4o and Claude 3.5 Sonnet frequently used an incorrect deadweight loss formula.
Limitations
- Due to budget and time constraints, closed-source models were evaluated on all elements while open-source models were run on only 20 elements and o1-preview on 13.
- Templates were generated with an LLM and hand-checked, so generation quality depends on LLM style-transfer consistency.
- Benchmark is restricted to multiple-choice format with a fixed set of microeconomic elements.
Experimental setup
Datasets: auto-STEER generated MCQA questions (STEER-ME)
Benchmarks: STEER-ME
Metrics: normalized accuracy exact-match accuracy expected calibration error Brier score domain robustness type robustness
Baselines: o1-preview claude-3-5-sonnet claude-3-haiku gpt-4o-mini
Builds on: STEER STEER Report Card zero-shot chain-of-thought
Closest papers by meaning
- STEER: Assessing the Economic Rationality of Large Language Models (ICML 2024 · similarity 0.90)
- FinBen: A Holistic Financial Benchmark for Large Language Models (NeurIPS 2024 · similarity 0.88)
- GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models (ICLR 2025 · similarity 0.86)
- The Shadow Price of Reasoning: Economic Perspective on Optimal Budget Allocation for LLMs (ICML 2026 · similarity 0.86)
- Training Language Models to Reason Efficiently (NeurIPS 2025 · similarity 0.86)
- Tailoring Self-Rationalizers with Multi-Reward Distillation (ICLR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.