Research map / Large language models / Language, safety and interpretability

Decision-Making Behavior Evaluation Framework for LLMs under Uncertain Context

NeurIPS 2024 · Large language models › Language, safety and interpretability · approach: Uncertainty quantification for LLMs · 6 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Evaluates risk preference, probability weighting, and loss aversion of ChatGPT-4-Turbo, Claude-3-Opus, and Gemini-1.0-pro using multiple-choice-list lottery experiments fitted with the TCN value-function model, then tests how embedded socio-demographic personas shift these parameters.

Problem

Whether LLM decision-making under uncertainty aligns with human behavioral-economics patterns (risk preference, probability weighting, loss aversion) and how socio-demographic persona embedding introduces biases.

Approach

Adapts the Tanaka-Camerer-Nguyen (TCN) prospect-theory-style utility model and revealed-preference switching-point estimation from human experiments to treat LLMs as respondents in lottery choice lists, then regresses the estimated parameters on demographic personas.

Key results

• All three LLMs show risk-averse behavior on average (sigma means: ChatGPT 0.60, Claude 0.31, Gemini 0.50), compared with a human sample mean of 0.48.
• Claude-3-Opus has the highest loss aversion (lambda mean 6.32) versus ChatGPT 1.48 and Gemini 2.33, with human sample 3.47.
• Embedding demographic personas significantly shifts parameters, e.g., Claude shows higher risk aversion for homosexual, bisexual, and asexual personas (coefficients around 0.044-0.057, p<0.001).

Measured effects reported in the paper

Surprising findings

• ChatGPT-4-Turbo's probability weighting alpha exceeds 1 (mean 1.18), diverging from the typical human overweighting of small probabilities.
• Claude-3-Opus's risk aversion increases for sexual minority and disability personas, contrary to the finding in some human studies of no significant differences.

Limitations

• Limited to three commercial LLMs and a financial lottery domain; authors recommend exploring other domains and models.
• Direct comparison with human subjects is difficult due to sensitivity of demographic features.
• Responses are sampled as repeated queries of a single session design, and the parameter estimation relies on midpoint interval estimates.

Experimental setup

Datasets: World Bank country demographic statistics

Benchmarks: Multiple-choice-list lottery experiments (3 series, 35 rows)

Metrics: risk preference sigma probability weighting alpha loss aversion lambda OLS regression coefficients

Baselines: Human sample estimates from prior behavioral economics studies

Builds on: Prospect Theory Expected Utility Theory TCN model (Tanaka, Camerer, Nguyen) Generalized Axiom of Revealed Preference

Closest papers by meaning

• Language Models Trained to do Arithmetic Predict Human Risky and Intertemporal Choice (ICLR 2025 · similarity 0.88)
• Large Language Models Assume People are More Rational than We Really are (ICLR 2025 · similarity 0.88)
• Distributive Fairness in Large Language Models: Evaluating Alignment with Human Values (NeurIPS 2025 · similarity 0.88)
• STEER: Assessing the Economic Rationality of Large Language Models (ICML 2024 · similarity 0.87)
• Can Large Language Models Serve as Rational Players in Game Theory? A Systematic Analysis (AAAI 2024 · similarity 0.87)
• Bridging Human and LLM Judgments: Understanding and Narrowing the Gap (NeurIPS 2025 · similarity 0.87)