Research map / Multi-agent systems and games / LLM agents
LieCraft: A Multi-Agent Framework for Evaluating Deceptive Capabilities in Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces LieCraft, a modular multiplayer hidden-role game with 10 realistic themes, used to measure LLM deception propensity, success, and accusation accuracy across 12 models.
Problem
Existing game-based deception evaluations (Among Us, Diplomacy, Avalon) rely on culturally embedded or gamified settings that may trigger memorized strategies or game-specific tropes, limiting real-world generalization of measured LLM deception.
Approach
A hidden-role card game whose mechanics are decoupled from its narrative, letting the same constraint-satisfied reward structure be re-themed into ten high-stakes domains, with models free to choose cooperator or defector roles, and measured via TrueSkill, defector selection rate, sabotage success, and difficulty-adjusted accusation scores.
Key results
- Claude 3.7 Sonnet ranks first on TrueSkill (33.7) with ~5% defector selection and 48% defector win rate.
- Gemma3-27B has the lowest TrueSkill (21.7) and struggles across dimensions.
- Over 1000 multiplayer games were run across 12 models and 10 themes; LLM-judge deception labels reached Cohen's kappa > 0.8 against blind human evaluation on 100 cases.
Surprising findings
- Models that are better at accusing defectors also tend to be better defectors, suggesting deception and detection skills develop together.
- Gemini-2.5-Flash and o4-mini rank in the bottom half of TrueSkill despite being proprietary models.
Limitations
- Game-based setting may still trigger game-playing behavior rather than real-world deception.
- Models under 20B parameters struggled to play the game; the largest models (Claude 4 Opus, o3, Gemini-2.5-Pro) were not evaluated due to cost.
- Some proprietary models' training data is unknown, limiting explanation of performance differences.
Experimental setup
Benchmarks: LieCraft
Metrics: TrueSkill mu/sigma defector selection rate defector win rate accusation score deception speech type proportions
Baselines: Among Us Diplomacy Avalon
Builds on: TrueSkill vLLM refusal direction abliteration
Closest papers by meaning
- Among Us: A Sandbox for Measuring and Detecting Agentic Deception (NeurIPS 2025 · similarity 0.91)
- DeceptionBench: A Comprehensive Benchmark for AI Deception Behaviors in Real-world Scenarios (NeurIPS 2025 · similarity 0.89)
- OpenDeception: Learning Deception and Trust in Human–AI Interaction via Multi-Agent Simulation (ICML 2026 · similarity 0.88)
- Incentivizing Truthful Language Models via Peer Elicitation Games (NeurIPS 2025 · similarity 0.87)
- Are Your Agents Upward Deceivers? (ICML 2026 · similarity 0.87)
- Evaluating Language Model Agency Through Negotiations (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.