Research map / Multi-agent systems and games / Human-AI and goal-conditioned
Sketchtopia: A Dataset and Foundational Agents for Benchmarking Asynchronous Multimodal Communication with Iconic Feedback
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Sketchtopia, a Pictionary-style dataset of 20K human gameplay sessions with sketches, guesses and iconic feedback, plus DrawBot and GuessBot agents evaluated with new asynchronous-communication metrics.
Problem
Lack of datasets and agents for goal-driven, asynchronous multimodal communication involving freehand sketches, intermediate guesses, and iconic feedback gestures in a cooperative Pictionary-style game.
Approach
A large human Pictionary dataset with timestamped strokes, erases, guesses and iconic feedback, and hierarchical agents (ActionDecider controlling a diffusion-based DrawBot and a CLIP retrieval-based GuessBot) that act asynchronously rather than in turns, benchmarked with new metrics (FRS, AAO, MATS).
Key results
- GUESSBOT achieves 81.7% win rate and 86.0% soft win rate versus 85.5% and 93.4% for human guessers.
- DRAWBOT achieves 51.6% win rate, markedly below the human 85.5%, with weak performance on adjectives (30%) and verbs (56%).
- GUESSBOT's feedback responsiveness (46%) is close to humans (40%), and its MATS (0.27) is closer to human pacing than DRAWBOT's (0.39).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DRAWBOT instead of a human Drawer | Sketchtopia human-agent gameplay · Sketchtopia test set (800 sessions) | Win rate: 85.5 -> 51.6 | Hurt |
| Use GUESSBOT instead of a human Guesser | Sketchtopia human-agent gameplay · Sketchtopia test set (800 sessions) | Win rate: 85.5 -> 81.7 | Hurt |
| Use GUESSBOT instead of a human Guesser | Sketchtopia human-agent gameplay · Sketchtopia test set (800 sessions) | Feedback Responsiveness Score: 40% -> 46% | No clear effect |
| Use GUESSBOT instead of a human Guesser | Sketchtopia human-agent gameplay · Sketchtopia test set (800 sessions) | Asynchronous Action Overlap: 0.29 -> 0.17 | Mixed |
| Use GUESSBOT instead of a human Guesser | Sketchtopia human-agent gameplay · Sketchtopia test set (800 sessions) | Multimodal Action Timing Similarity: 0.00 -> 0.27 | Helped |
| Use DRAWBOT instead of a human Drawer | Sketchtopia human-agent gameplay · Sketchtopia test set (800 sessions) | Multimodal Action Timing Similarity: 0.00 -> 0.39 | Hurt |
Surprising findings
- Off-the-shelf VLMs produced verbose and unhelpful guesses and were too slow for asynchronous play, so they were judged unviable as guessers.
- GUESSBOT's retrieval-based guesses reproduce human-like spelling errors, which the authors frame as a realism feature.
Limitations
- Dataset skews toward younger, right-handed, English-speaking participants from a limited region.
- Iconic feedback is limited to a predefined set of icons (-, ,, å).
- DrawBot performs poorly on adjectives and verbs, which lack direct object referents.
Experimental setup
Datasets: Sketchtopia
Benchmarks: Sketchtopia
Metrics: Win Rate Soft Win Rate Feedback Responsiveness Score Asynchronous Action Overlap Multimodal Action Timing Similarity
Baselines: Human-Human games VLMs as guessers (qualitative)
Builds on: Stable Diffusion ControlNet T2I-Adapter CLIP Annoy
Closest papers by meaning
- SketchAgent: Language-Driven Sequential Sketch Generation (CVPR 2025 · similarity 0.88)
- EgoBench: An Interactive Egocentric Multimodal Benchmark for Tool-Using Agents (NeurIPS 2026 · similarity 0.87)
- ArtWhisperer: A Dataset for Characterizing Human-AI Interactions in Artistic Creations (ICML 2024 · similarity 0.86)
- PARTNR: A Benchmark for Planning and Reasoning in Embodied Multi-agent Tasks (ICLR 2025 · similarity 0.86)
- Steve-Eye: Equipping LLM-based Embodied Agents with Visual Perception in Open Worlds (ICLR 2024 · similarity 0.86)
- BEAR: Dissecting Embodied Abilities in Multimodal Language Models through Skill-level Evaluation and Diagnosis (ICML 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.