Research map / Learning theory and algorithms / Prediction and decision losses
How to Evaluate Behavioral Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Axiomatically characterizes which loss functions are suitable for evaluating behavioral models over discrete outcome distributions, and shows that diagonal bounded Bregman divergences satisfy all axioms, with squared L2 error the only common loss in this class.
Problem
Behavioral game theory and related fields lack agreement on which loss function to use when evaluating predictive but interpretable models of human behavior from multiple samples over discrete outcomes (error rate, NLL, cross-entropy, Brier score, squared L2 error are all used).
Approach
Formalizes five axioms (sample and distributional Pareto-alignment, empirical distribution sufficiency, counterfactual Pareto-regularity, zero minimum) for evaluation losses, and identifies diagonal bounded Bregman divergences as the family satisfying all of them; squared L2 error is the common loss in this family, while scoring rules fail the interpretability axioms.
Key results
- Squared L2 error is the only commonly used loss that is a diagonal bounded Bregman divergence and satisfies all five axioms; every other common loss violates at least one.
- Error rate is minimized by predicting the mode, so it violates sample propriety and zero-minimum (example with 6/10 defectors gives error 0.4 for f=1 vs 0.48 for the empirical distribution).
- No scoring rule satisfies counterfactual Pareto-regularity, and every scoring rule satisfying sample Pareto-alignment violates zero-minimum.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Evaluate the same empirical distribution with 20 rather than 10 observations using NLL | Negative log-likelihood · Same empirical distribution (0.6, 0.4) | loss invariance to sample count (EDS): 2.9 -> 5.8 | Hurt |
| Use Brier score to evaluate the original empirical-distribution prediction on the third dataset | Brier score · Third dataset; 9 defecting and 1 cooperating | loss under counterfactual data change (CPR): 0.48 -> 0.36 | Hurt |
| Use cross-entropy to evaluate the original empirical-distribution prediction on the third dataset | Cross-entropy · Third dataset; 9 defecting and 1 cooperating | loss under counterfactual data change (CPR): 0.29 -> 0.24 | Hurt |
| Predict the mode rather than the empirical distribution under error rate | Error rate · 10 participants; empirical distribution (0.6, 0.4) | sample propriety: 0.48 -> 0.4 | Hurt |
| Predict the median empirical probability, 0.7, rather than the true probability, 2/3, under MAE | MAE · 10 observations sampled from (2/3, 1/3) | expected MAE: 0.243 -> 0.235 | Mixed |
| Use squared L2 error rather than the other common losses as the evaluation loss | Behavioral-model evaluation · Finite discrete outcomes; general | satisfaction of the stated axioms | Helped |
| Use MAE rather than a loss satisfying distributional Pareto-alignment | Behavioral-model evaluation · Finite discrete outcomes; general | distributional Pareto-alignment | Hurt |
| Use NLL rather than a loss satisfying empirical distribution sufficiency | Behavioral-model evaluation · Same empirical distribution with different sample counts | empirical distribution sufficiency | Hurt |
| Use KL divergence when predictions assign zero probability to observed actions | Behavioral-model evaluation · Three-action dataset with empirical distribution (0.01, 0.19, 0.80) | sample Pareto-alignment | Hurt |
| Use a scoring rule for model-performance reporting rather than its corresponding DBBD | Scoring rules · Finite discrete outcomes; general | counterfactual Pareto-regularity | Hurt |
Surprising findings
- Every scoring rule satisfying sample Pareto-alignment violates zero-minimum, so scoring rules are unsuitable for reporting model performance despite their popularity.
- Error rate, MAE and other common losses can rank a non-perfect prediction above the true distribution in expectation.
Limitations
- Axioms (e.g., zero-minimum, DPA) are subjective and may not suit all settings; zero-minimum could be replaced by a uniform-baseline anchor.
- Interpretability axioms are poorly motivated for continuous distributions or single-observation settings.
- Does not identify a single best loss; additional axioms (e.g., action-label agnosticism) could narrow the family further.
Experimental setup
Datasets: experimental data on two traveler's dilemma games (from prior work)
Benchmarks: traveler's dilemma experimental data
Metrics: error rate MAE negative log-likelihood cross-entropy KL divergence Brier score squared L2 error
Baselines: error rate mean absolute error negative log-likelihood cross-entropy KL divergence Brier score
Builds on: proper scoring rules property elicitation (Abernethy et al. 2012) Bregman divergences
Closest papers by meaning
- Quantifying the Sensitivity of Inverse Reinforcement Learning to Misspecification (ICLR 2024 · similarity 0.83)
- Decision-Making Behavior Evaluation Framework for LLMs under Uncertain Context (NeurIPS 2024 · similarity 0.82)
- Integrating Inference and Experimental Design for Contextual Behavioral Model Learning (AAAI 2025 · similarity 0.82)
- Model Alignment as Prospect Theoretic Optimization (ICML 2024 · similarity 0.82)
- Language Models Trained to do Arithmetic Predict Human Risky and Intertemporal Choice (ICLR 2025 · similarity 0.81)
- A Fine-grained Analysis of Fitted Q-evaluation: Beyond Parametric Models (ICML 2024 · similarity 0.81)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.