Research map / Multi-agent systems and games / Human-AI and goal-conditioned
SOTOPIA: Interactive Evaluation for Social Intelligence in Language Agents
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces SOTOPIA, an open-ended interactive environment with 90 scenarios and 40 characters for simulating goal-driven social interactions, plus SOTOPIA-EVAL, a seven-dimension evaluation framework validated against human judgments with GPT-4 as evaluator.
Problem
Existing social intelligence benchmarks for AI are mostly static and non-interactive, and existing interactive evaluations do not cover diverse goal-driven, multi-dimensional social behavior.
Approach
An LLM-generated, procedurally sampled task space (scenarios, goals, characters, relationships) for dyadic role-play, scored along seven social dimensions (goal completion, believability, knowledge, secret keeping, relationship, social rules, financial benefits), with GPT-4 used as an automated evaluator calibrated against human ratings.
Key results
- GPT-4 scores highest on most SOTOPIA-EVAL dimensions among the tested models (e.g., Goal 7.62 vs 6.45 for GPT-3.5, 5.38 for Llama-2-70b-chat, 4.10 for MPT-30b-chat).
- GPT-4 correlates with human judgments most strongly on goal completion when models role-play (r=0.71), and on Fin and Rel as well, but correlations drop when humans role-play.
- On SOTOPIA-hard, humans achieve significantly higher goal completion than GPT-4 (6.15 with humans vs 4.85 for GPT-4 with humans).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use GPT-4 instead of GPT-3.5 | SOTOPIA agents · SOTOPIA | Goal Completion: 6.45 -> 7.62 | Helped |
| Use GPT-4 instead of GPT-3.5 | SOTOPIA agents · SOTOPIA | Believability: 9.15 -> 9.28 | Helped |
| Use GPT-4 instead of GPT-3.5 | SOTOPIA agents · SOTOPIA | Knowledge: 3.40 -> 3.73 | Helped |
| Use GPT-4 instead of GPT-3.5 | SOTOPIA agents · SOTOPIA | Relationship: 1.23 -> 1.94 | Helped |
| Use GPT-4 instead of GPT-3.5 | SOTOPIA agents · SOTOPIA | Financial and Material Benefits: 0.46 -> 0.81 | Helped |
| Use GPT-4 instead of GPT-3.5 | SOTOPIA agents · SOTOPIA | Social Rules: -0.08 -> -0.07 | Helped |
| Use GPT-4 instead of GPT-3.5 | SOTOPIA agents · SOTOPIA | Secret: -0.08 -> -0.14 | Hurt |
| Use humans instead of GPT-4 | SOTOPIA-hard agents interacting with humans · SOTOPIA-hard | Goal Completion: 4.85 -> 6.15 | Helped |
| Use humans instead of GPT-4 | SOTOPIA-hard agents interacting with GPT-4 · SOTOPIA-hard | Goal Completion: 4.85 -> 5.95 | Helped |
Surprising findings
- All models get negative scores on secret keeping and social rules, with GPT-4 not better than other models on these dimensions.
- Llama-2-70b-chat scores lower than GPT-3.5 in interactive settings, diverging from static benchmark rankings.
Limitations
- Dyadic interactions only, five relationship types, and a limited pool of 40 characters and 90 scenarios.
- GPT-4 evaluator correlates weakly with humans on several dimensions when humans role-play and shows possible positional and other biases.
- Human-model comparison limited to 20 hard scenarios due to resource constraints.
Experimental setup
Datasets: Social Chemistry Social IQa Deal-or-no-Deal NormBank CraigslistBargains MutualFriends
Benchmarks: SOTOPIA
Metrics: goal completion believability knowledge secret keeping relationship social rules financial and material benefits Pearson correlation
Baselines: GPT-3.5 Llama-2-70b-chat MPT-30b-chat
Builds on: Social Chemistry Social IQa CraigslistBargains Dec-POMDP
Closest papers by meaning
- Theory of Mind and Persuasion Beyond Conversation: Assessing the Capacity of LLMs to Induce Belief States via Planning and Action (NeurIPS 2026 · similarity 0.86)
- Evaluating Generalization Capabilities of LLM-Based Agents in Mixed-Motive Scenarios Using Concordia (NeurIPS 2025 · similarity 0.86)
- $\tau^2$-Bench: Evaluating Conversational Agents in a Dual-Control Environment (ICML 2026 · similarity 0.86)
- Implicit Intelligence - Evaluating Agents on What Users Don’t Say (ICML 2026 · similarity 0.85)
- ALSO: Adversarial Online Strategy Optimization for Social Agents (ICML 2026 · similarity 0.85)
- MetaMind: Modeling Human Social Thoughts with Metacognitive Multi-Agent Systems (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.