Research map / Large language models / AI and society

Towards Real-world Human Behavior Simulation: Benchmarking Large Language Models on Long-horizon, Cross-scenario, Heterogeneous Behavior Traces

NeurIPS 2026 · Large language models › AI and society · approach: Data commons and human baselines · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces OmniBehavior, a real-world Kuaishou user-log benchmark for LLM user simulation spanning long horizons, five scenarios and 22 actions, and reports that LLM simulators underperform and show positivity-and-average biases.

Problem

Existing user-simulation benchmarks are confined to isolated scenarios, narrow action spaces, or synthetic data, so they fail to test LLMs on long-horizon, cross-scenario human behavior.

Approach

A benchmark built from authentic multi-scenario industrial logs (200 users, 3 months, 2.12M interactions, 22 actions) with a user-conditioned prediction task, plus analyses of cross-scenario causal chains and a "positivity-and-average" structural bias in LLM simulators.

Key results

• Claude-Opus-4.5 is the best model with an overall score of 44.55; most models fall between 32 and 41.
• Binary F1 for like/share-type behaviors is mostly below 40% across models.
• Memory summarization gives the best average binary F1 (24.27 vs 21.13 truncation) on Qwen3-235B, while RAG underperforms truncation (20.38).

Surprising findings

• Performance plateaus or does not consistently improve as context extends from 16K to 128K tokens.
• Open-weight GLM-4.7 ranks second overall (41.46), ahead of Claude-Sonnet-4.5 and GPT-5.2.

Limitations

• Evaluated models and memory strategies do not cover all state-of-the-art approaches.
• Benchmark covers only 200 users from a single platform (Kuaishou).

Experimental setup

Datasets: OmniBehavior (Kuaishou logs) LoCoMo

Benchmarks: OmniBehavior

Metrics: F1 NMAE LLM-judge score overall score positive interaction rate intra/inter-user distance ratio

Baselines: Claude-Opus-4.5 Claude-Sonnet-4.5 Claude-Haiku-4.5 Claude-Sonnet-4 Gemini-3-Flash GPT-5.2 GPT-4o GLM-4.7 DeepSeek-V3 Kimi-K2-Instruct Qwen3-235B

Builds on: LoCoMo

Closest papers by meaning

• OdysSim: Building Foundation Models for Human Behavior Simulation (NeurIPS 2026 · similarity 0.93)
• Measuring and Mitigating the Distributional Gap Between Real and Simulated User Behaviors (NeurIPS 2026 · similarity 0.90)
• Position: LLM Social Simulations Are a Promising Research Method (ICML 2025 · similarity 0.88)
• CogBench: a large language model walks into a psychology lab (ICML 2024 · similarity 0.88)
• LLM Generated Persona is a Promise with a Catch (NeurIPS 2025 · similarity 0.87)
• BehaviorBench: Modeling Real-World User Decisions from Behavioral Traces (NeurIPS 2026 · similarity 0.87)