Research map / Multi-agent systems and games / Human-AI and goal-conditioned

Playing ZendoWorld: Challenging AI Agents on Active Visual Concept Induction

NeurIPS 2026 · Multi-agent systems and games › Human-AI and goal-conditioned · approach: Human-AI coordination agents · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces ZendoWorld, a 3D visual active rule-induction benchmark (Zendo-style) and evaluates VLM, Bayesian SMC, VLP and neuro-symbolic oracle agents plus a human study, finding labeling accuracy dissociates from rule recovery and VLM agents propose low-information experiments.

Problem

Existing inductive reasoning benchmarks are passive or symbolic, so it is unclear whether agents can jointly perceive visual scenes, induce hidden rules, and design informative experiments in an interactive loop.

Approach

An interactive Zendo-inspired environment with Blender-rendered 3D scenes, a Prolog-based DSL rule space, and a loop where agents propose scenes, receive labels, guess rules, and get counterexamples; diagnostics include structural F1 and expected information gain.

Key results

• Oracle Agent solves 95.5% of games in 4.5 turns on symbolic input, but fails the OOD same-amount rule; VLM Agent solves 44.5% visually (GPT-5-mini backbone), Bayesian 13.6%, VLP 18.2%.
• Removing visual input raises VLP wins from 18.2% to 48.5% and VLM wins to 57.6%, indicating perception is a bottleneck for some agents but not others.
• VLM and Bayesian agents show near-zero expected information gain after about 10 examples, while humans solved 7 of 10 participants on the OOD task where no visual agent succeeded.

Surprising findings

• Bayesian Agent achieves the highest label accuracy (75.2%) yet solves few games, showing label accuracy does not imply rule recovery.
• Oracle Agent with random experiments achieves the second-highest EIG, outperforming hypothesis-guided VLM and Bayesian agents.

Limitations

• Oracle agent is only effective when the true rule is expressible in its DSL, failing on the OOD rule.
• Human study has only 19 participants and a subset of games.
• Fallback LLM judge is used for rule equivalence in a small fraction of guesses (1.8-6.3%).

Experimental setup

Datasets: synthetic Blender-rendered Zendo scenes Looney Labs Zendo rule set (928 rules)

Benchmarks: ZendoWorld (22 games) Symbolic ZendoWorld

Metrics: win rate average turns label accuracy structural F1 expected information gain

Baselines: VLM Agent Bayesian Agent (LLM-SMC) Vision-Language Programs Agent Oracle Agent (Random) Human participants

Builds on: Zendo-based setups LLM-SMC DeepSynth Vision Language Programs Science-Gym

Closest papers by meaning

• VAGEN: Reinforcing World Model Reasoning for Multi-Turn VLM Agents (NeurIPS 2025 · similarity 0.87)
• GAM-Agent: Game-Theoretic and Uncertainty-Aware Collaboration for Complex Visual Reasoning (NeurIPS 2025 · similarity 0.87)
• PhysVLM-AVR: Active Visual Reasoning for Multimodal Large Language Models in Physical Environments (NeurIPS 2025 · similarity 0.86)
• Agentic Jigsaw Interaction Learning for Enhancing Visual Perception and Reasoning in Vision-Language Models (ICLR 2026 · similarity 0.86)
• What if Agents Could Imagine? Reinforcing Open-Vocabulary HOI Comprehension through Generation (NeurIPS 2026 · similarity 0.86)
• Concept-RuleNet: Grounded Multi-Agent Neurosymbolic Reasoning in Vision Language Models (AAAI 2026 · similarity 0.85)