Research map / Large language models / AI and society

Measuring and Mitigating the Distributional Gap Between Real and Simulated User Behaviors

NeurIPS 2026 · Large language models › AI and society · approach: Data commons and human baselines · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes a method to measure the distributional gap between real and LLM-simulated user behaviors (LLM-extracted behavior descriptions, embedding, k-means quantization, KL/JS divergences) and evaluates 24 user simulators on WildChat coding and writing conversations.

Problem

User simulators are trained to produce human-like responses, but whether they capture the broad, heterogeneous distribution of real user behaviors is unmeasured.

Approach

Represent each conversation by LLM-generated descriptions of user behavior along six facets, embed them, quantize with k-means into discrete behavior distributions for real and simulated users, and measure gap via forward/backward KL and JS divergence (mapping to recall and precision failures); also show combining complementary simulators narrows the gap.

Key results

• Gemini 3.1 Pro and GPT-5.4 achieve the smallest gaps (coding JS about 0.25, writing JS about 0.47-0.48), while all simulators show a large gap from real users.
• Combining Gemini 3.1 Pro with GPT-5.4 lowers coding JS to 0.226 from 0.246 for the better individual simulator.
• Odd-one-out human study: annotators identified the odd behavior description 86.7% of the time (Fleiss' kappa 0.74); linear classifiers separate real from simulated embeddings with 90.92% to 99.63% accuracy.

Surprising findings

• Scale alone does not close the gap: Llama-3.1-8B-Instruct outperforms Llama-3.3-70B-Instruct, and gemma-4-31B-it and gpt-oss-120b are competitive with several closed-source models.
• Simpler representations (raw conversations, user utterances only) yield low divergences for all simulators, masking the gap that behavior descriptions reveal.

Limitations

• Real conversations were collected from WildChat with ChatGPT/GPT-4 assistants, while simulated conversations used Qwen3.5-122B-A10B as assistant, confounding assistant effects.
• Relies on a capable LLM (Qwen3.5-122B-A10B-FP8) to generate accurate behavior descriptions; user goals extracted by GPT-4o may be lossy.
• Only English conversations and coding/writing tasks; WildChat population skews toward IT/developers; UserLM-8b was trained on WildChat, overlapping with evaluation data.

Experimental setup

Datasets: WildChat WildChat postprocessed subset with GPT-4o user goals

Benchmarks: WildChat coding subset (5,000 conversations) WildChat writing subset (5,000 conversations)

Metrics: forward KL backward KL Jensen-Shannon divergence MAUVE nearest neighbor cosine similarity linear classification accuracy

Baselines: UserLM-8b humanlm-opinion GPT-5.4 Gemini 3.1 Pro Claude Haiku 4.5 Llama-3.3-70B-Instruct

Builds on: precision and recall for generative models MAUVE DAMSL Schema-Guided Dialogue

Closest papers by meaning

• Towards Real-world Human Behavior Simulation: Benchmarking Large Language Models on Long-horizon, Cross-scenario, Heterogeneous Behavior Traces (NeurIPS 2026 · similarity 0.90)
• OdysSim: Building Foundation Models for Human Behavior Simulation (NeurIPS 2026 · similarity 0.89)
• Non-Collaborative User Simulators for Tool Agents (ICLR 2026 · similarity 0.86)
• BehaviorBench: Modeling Real-World User Decisions from Behavioral Traces (NeurIPS 2026 · similarity 0.85)
• First-Person Fairness in Chatbots (ICLR 2025 · similarity 0.85)
• Model-Free Assessment of Simulator Fidelity via Quantile Curves (NeurIPS 2026 · similarity 0.85)