Research map / Large language models / AI and society

BehaviorBench: Modeling Real-World User Decisions from Behavioral Traces

NeurIPS 2026 · Large language models › AI and society · approach: Data commons and human baselines · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces BehaviorBench, a benchmark of wallet-level prediction-market decision histories reconstructed from public on-chain logs, evaluating Belief and Trade prediction under no-history, direct-history, profile, and retrieval interfaces across frontier and open-weight LLMs.

Problem

Evaluating personalized decision modeling is hard because user preferences and beliefs are rarely stated, and simulated or persona-based users may not reflect real human behavior.

Approach

Builds a benchmark from real public prediction-market on-chain traces with two layers (final revealed side/confidence per wallet-market, and next trade direction/amount), and treats the history representation (direct recent history, generated profile, retrieved support-wallet evidence) as part of the evaluation.

Key results

• On Belief prediction, gpt-5.4 with ProfileGen reaches 73.20% choice accuracy vs. 53.13% for no personalization.
• On Trade prediction, gpt-5.4 with DirectGen reaches 75.38% direction accuracy vs. 52.85% for no personalization.
• ProfileGen gives the best Belief confidence error for several models (e.g., claude-3.7-sonnet 0.0783), while DirectGen is strongest for Trade direction and amount for many models.

Surprising findings

• Trade direction accuracy and amount error favor raw recent history over compressed profiles and retrieval, and random profiles can produce very large amount errors.
• Belief accuracy generally declines in wallets with more markets, so broader history is not automatically easier to exploit.

Limitations

• Wallets are pseudonymous and may not correspond to single real people; labels reflect revealed behavior, not private beliefs.
• Trade amounts may depend on unobserved portfolio constraints, liquidity, fees, or strategy.
• Main benchmark is restricted to data before block 80,000,000 (December 2025) to avoid a regime shift, limiting temporal coverage.

Experimental setup

Datasets: Polymarket public API metadata Polygon on-chain logs

Benchmarks: BehaviorBench Belief prediction BehaviorBench Trade prediction

Metrics: Choice Accuracy Confidence MAE Direction Accuracy Median Absolute Error

Baselines: No Personalization

Closest papers by meaning

• PANTHER: Generative Pretraining Beyond Language for Sequential User Behavior Modeling (NeurIPS 2025 · similarity 0.88)
• Towards Real-world Human Behavior Simulation: Benchmarking Large Language Models on Long-horizon, Cross-scenario, Heterogeneous Behavior Traces (NeurIPS 2026 · similarity 0.87)
• Measuring and Mitigating the Distributional Gap Between Real and Simulated User Behaviors (NeurIPS 2026 · similarity 0.85)
• Action-On-Item Preference Flow: A Shared Event Schema for Predictive and Generative Personalization (NeurIPS 2026 · similarity 0.85)
• Listwise Preference Diffusion Optimization for User Behavior Trajectories Prediction (NeurIPS 2025 · similarity 0.85)
• Large Language Models Assume People are More Rational than We Really are (ICLR 2025 · similarity 0.85)