Research map / Multi-agent systems and games / LLM agents
WhodunitBench: Evaluating Large Multimodal Agents via Murder Mystery Games
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces WhodunitBench, a murder-mystery-game benchmark for large multimodal agents with an arena-style win-rate evaluation over 50 scripts and a Chain of Evaluation of over 3,000 multiple-choice and open-ended questions.
Problem
Existing benchmarks for large multimodal agents focus on single tasks or static scenarios and do not assess compositional skills (multimodal perception, interaction, multi-step reasoning, decision making) in dynamic, incomplete-information settings.
Approach
Uses murder mystery games as a simulated dynamic environment: agents play roles in pairwise faction-based arena matches (win rate) and are also scored on fine-grained capabilities (perception QA, role-play, decision making, multi-step cognition) via a Chain of Evaluation built from annotated scripts and clues.
Key results
- In the arena, all evaluated LMAs achieved win rates between roughly 10% and 25% against peers; GPT-4V had the highest average win ratio at 24.2%.
- On the Chain of Evaluation, GPT-4V scored 93.60 on long-script QA and 79.29 on text-rich image QA with direct prompting, but only about 20 on the discussion-phase scores (SPC/ITD).
- Chain-of-thought did not uniformly help: for GPT-4V it lowered text-rich image QA from 79.29 to 51.88 while raising multi-step reasoning slightly.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use GPT-4V rather than Qwen-VL-Plus | WhodunitBench arena agents · WhodunitBench arena | average win rate: 10.6% -> 24.2% | Helped |
| Use COT instead of Direct prompting (ablation) | GPT-4V · WhodunitBench LSU | score: 93.60 -> 92.40 | Hurt |
| Use COT instead of Direct prompting (ablation) | GPT-4V · WhodunitBench TIU | score: 79.29 -> 51.88 | Hurt |
| Use COT instead of Direct prompting (ablation) | GPT-4V · WhodunitBench MIU | score: 68.09 -> 69.25 | Helped |
| Use COT instead of Direct prompting (ablation) | GPT-4V · WhodunitBench RP | score: 9.49 -> 6.43 | Hurt |
| Use COT instead of Direct prompting (ablation) | GPT-4V · WhodunitBench SPC | score: 9.49 -> 19.63 | Helped |
| Use COT instead of Direct prompting (ablation) | GPT-4V · WhodunitBench ITD | score: 25.95 -> 16.28 | Hurt |
| Use COT instead of Direct prompting (ablation) | GPT-4V · WhodunitBench MMR | score: 57.12 -> 58.75 | Helped |
| Use COT instead of Direct prompting (ablation) | GPT-4V · WhodunitBench CMD | score: 25.18 -> 26.43 | Helped |
| Use COT instead of Direct prompting (ablation) | GPT-4V · WhodunitBench average | score: 45.84 -> 42.63 | Hurt |
Surprising findings
- Stronger models (e.g., Gemini) are not necessarily better when playing the murderer; they tend to over-communicate and get identified more often.
- Chain-of-thought sometimes reduces performance, especially on tasks like direct text recognition from images.
Limitations
- Evaluation is expensive because scripts often exceed 5,000 words per character and API usage is large.
- Social interaction and reasoning are entangled, making results harder to interpret.
- Dataset is small (50 scripts) and not suited for large-scale training.
Experimental setup
Datasets: 50 murder mystery scripts with image and text clues
Benchmarks: WhodunitBench
Metrics: win rate loss rate TIU accuracy MIU accuracy LSU accuracy RP score SPC score key character questioning score MMR accuracy CMD score
Baselines: Yi-Vision Qwen-VL-Plus Gemini-pro-vision Claude-Opus GPT-4V Naive Agent
Builds on: chain-of-thought
Closest papers by meaning
- AgentBench: Evaluating LLMs as Agents (ICLR 2024 · similarity 0.87)
- Multi-Agent Undercover Gaming: Hallucination Removal Through Counterfactual Test for Multimodal Reasoning (AAAI 2026 · similarity 0.86)
- SkillsBench: Benchmarking How Well Agent Skills Work Across Diverse Tasks (NeurIPS 2026 · similarity 0.86)
- GameArena: Evaluating LLM Reasoning through Live Computer Games (ICLR 2025 · similarity 0.86)
- GlitchBench: Can Large Multimodal Models Detect Video Game Glitches? (CVPR 2024 · similarity 0.86)
- BALROG: Benchmarking Agentic LLM and VLM Reasoning On Games (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.